AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

feature-usage-feed

Set up an LLM-judge evaluation that extracts canonical use cases for a PostHog feature at scale and streams the results to a Slack channel as a live feed. Use when someone wants to understand how users are actually using a specific AI/LLM-powered feature in production — what they're investigating, what questions they're trying to answer, and what patterns surface — without manually reading hundreds of traces. Assumes the feature emits `$ai_generation` and `$ai_evaluation` events with `$session_id` linkage to the trigger user's recording (the standard setup post the session-summary linkage PRs).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--feature-usage-feed at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 1

SKILL.md · 24000

This is the test that tells you "it works" for real. If no matching real event exists yet (common if the eval was just enabled), trigger the feature yourself, wait ~1 minute, and retry.

Recommended flow: synthetic → sanity-check the block template renders → flip real-requests on → confirm an actual post lands in the channel → save + enable the workflow.

Step 7 — End-to-end verify in production

Once the workflow is enabled, trigger the feature yourself. Within a minute or two:

  1. The $ai_generation event should appear in AI observability
  2. The eval should auto-run and emit an $ai_evaluation event
  3. The workflow should fire and the Slack post should land in the configured channel
  4. Click "View Trigger Session" — should land on the recording of you using the feature, not the replay homepage

If "View Trigger Session" lands on the replay homepage, $session_id is missing on the $ai_evaluation event (which is separate from the $ai_generation event — threading is independent for the two). Backend fix needed — see prerequisites.

Worked example A (Pattern A): group session summary use cases

Pattern: a group_summary_use_case_feed eval streaming to a #<team>-usage-feed channel. Trace prefix: session-summary:group:. Opener: "A user ran a group summary on". Slack channel showed e.g.:

📊 group_summary_use_case_feed triggered by some user "A user ran a group summary on a company's onboarding sessions from the last 7 days. They were trying to understand why account activation rates are low. The summary surfaced that most users abandon at the company onboarding wizard after creating accounts." [View Trace] [View Trigger Session] [View Person]

The PRs that made this work (linked here as worked examples of the session_id threading pattern, not as steps in the skill itself):

  • PostHog/posthog#54952 — threads trigger_session_id through to $ai_generation events on the session summary backend
  • (Followup PR — threads $session_id onto $ai_evaluation events specifically)

Worked example B (Pattern B): PostHog AI in error tracking mode

Pattern: an agent_mode = 'error_tracking' scoped feed streaming to a #<team>-usage-feed channel, answering "what are users actually trying to DO when they chat with PostHog AI in error tracking mode?" Mode sizing varies by an order of magnitude or more across agent modes — spot-check volume per §Step 1 before wiring, because a high-volume mode can flood a channel. Opener: "A user asked PostHog AI about".

Enabling PR: PostHog/posthog#55160 — threads agent_mode and supermode onto every $ai_generation emitted by the chat agent loop. Wiring lives in ee/hogai/core/agent_modes/executables.py (AgentExecutable._get_model) and passes the dict through the existing posthog_properties field on MaxChatMixin in ee/hogai/llm.py. Before this PR, scoping a PostHog AI eval to a specific mode wasn't possible — you'd end up evaluating every PostHog AI generation, which produced noisy feeds with low single-digit PASS rates.

Key observation from setup: the agent_mode tag reflects the mode at turn-time, but chat state retains mode selection even when users drift off-topic mid-conversation. Spot-check: a random agent_mode=error_tracking sample included a conversation that ended up being about session replay pricing. The eval prompt's classification must be permissive about topic drift — PASS only when the turn is recognizably in-scope for the mode, FAIL when the conversation has drifted to something else entirely.

Validating signal quality after launch

Once the feed has been running for a day or two, sanity-check the eval output at scale.

If posthog:llma-evaluation-report-create is exposed: schedule an AI report on the eval so the pass/fail picture keeps arriving without you asking for it.

posthog:llma-evaluation-report-create
{
  "evaluation": "<uuid>",
  "frequency": "scheduled",
  "rrule": "FREQ=WEEKLY;BYDAY=MO",
  "delivery_targets": [{ "type": "email", "value": "[email protected]" }]
}

posthog:llma-evaluation-report-generate runs a configured report right away, and posthog:llma-evaluation-report-run-list returns the content of past runs.

UI fallback: open the eval in AI observability → Evaluations, then the "Reports" tab.

If the FAIL bucket is large, the classification step is too strict — relax it. If the PASS bucket has lots of generic reasonings, iterate on the prompt to enforce concreteness. The report gives a quick read on this without you having to scroll through individual events.

Spot-check raw events when needed (note: the stored result value is 'True', not 'PASS' — see step 6):

SELECT
    properties.$ai_evaluation_reasoning AS reasoning,
    properties.$ai_trace_id AS trace_id,
    timestamp
FROM events
WHERE event = '$ai_evaluation'
    AND properties.$ai_evaluation_name = '<your eval name>'
    AND properties.$ai_evaluation_result = 'True'
    AND timestamp > now() - INTERVAL 1 DAY
ORDER BY timestamp DESC
LIMIT 25

Tips

  • The reasoning field IS the Slack message — design the prompt for that, not for "chain of thought before classification." Models can produce structured Slack-ready text in one pass.
  • LLM judges are non-deterministic across reruns. Expect 1-5% noise even with a fixed prompt and model. If you need reproducibility, pin a deterministic provider/seed in model_configuration.
  • Keep the eval scoped tightly via the conditions property filters on $ai_trace_id prefix. Otherwise it fans out to every $ai_generation event in the project and burns LLM cost.
  • For high-volume features (>10k traces/week), consider sampling — set the eval to run on a percentage of matching events rather than all of them. Slack flooding is a real failure mode.
  • The "View Trigger Session" button is the highest-value link in the alert. Without it, the feed is just text — you can't watch what the user was actually doing. Verify it works in step 7 before considering the feed shipped.
  • Once the feed is live, keep reading the passing runs' reasoning (a scheduled report, or the SQL above with $ai_evaluation_result = 'True') to surface the dominant use case clusters. That's how you turn the feed into actual product insights instead of just a notification stream.