AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

signals-scout-product-analytics

Signals scout for core product-analytics flows — funnels, retention, lifecycle, stickiness, and paths. Watches the team's saved flows for a derived-rate regression while entrants hold.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--signals-scout-product-analytics at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 2

SKILL.md · 24000

eport is no longer live, author a fresh report for the relapse and repoint that rate's report:product_analytics:flow:<short_id>:<rate> entry at the new id.

  • Author a fresh report via scout-emit-report when nothing in the inbox covers it (or a known regression has new evidence that changes the verdict). A strong finding here: the rate dropped clearly below the flow's seasonality-matched baseline (robust z ≥ ~3, or a conversion-point drop beyond the baseline band), the entrant denominator held (quantify both — "step-2 conversion 62%→48% while step-1 entrants steady at ~5.2k/day"), the move is broad across segments (not one known cohort), and it's not explained by a running experiment or a flow-definition edit. Put the flow short_id, the latest-window rate, the baseline band, the per-step/per-cohort numbers, the entrant volumes, and the time window in the evidence. A rate regression is chart-shaped — attach the regressed rate via charts, matched to the flow type: a funnel or retention rate with its steady entrant denominator alongside (the drop-on-steady-entrants shape), lifecycle as the state composition over time, stickiness as the frequency distribution against its population. A behavioral regression is an investigation with a starting point, not a question for a person: the flow, the step that moved, and the onset are where an agent begins, so set actionability=immediately_actionable, set priority, and name the repository that owns the flow (omit the field when you can't tell which repo that is, so selection can find it). Keep actionability=requires_human_input for a move whose next step is a product call — whether the new behavior was intended, which of several valid flows should win — and on those leave priority and repository unset, since they're PR-autostart fields and supplying priority + suggested_reviewers with no repository signals PR intent that spins up a repo-selection sandbox only to no-op (autostart needs immediately_actionable). Reach for them (P2 broad regression on a human-saved flow, P3 single-segment / inferred) only on the rare regression you'd actually want a draft PR for. Set suggested_reviewers whenever you can confidently resolve one — each entry is {github_login?, user_uuid?}, and the usual route here is to pass the flow's owning person as a user_uuid, or reuse a cached reviewer:product_analytics:<area> login. user_uuid resolution is fail-loud: a created_by that is not an org member rejects the whole emit-report, not only the reviewer. Do not pass a raw created_by if you cannot verify membership. Prefer a cached login or a created_by that routed before. You can also author the report without a reviewer and add one later. scout-members-list gives this project's members with their user_uuid and resolved github_login. Routing helps the report reach a human, so resolve an owner when you can. After authoring, write a rate-scoped report:product_analytics:flow:<short_id>:<rate> scratchpad entry with the report_id. Use a separate entry for each rate. The harness prompt carries the full report-channel contract.
  • Remember if suggestive but below that bar, or to refresh a baseline.
  • Skip if a noise: / addressed: / dedupe: entry, or an existing inbox report, already covers it.

If anomaly-detection already owns a related metric move in the inbox, author only if your behavioral-rate angle is materially new; otherwise edit-or-skip. The same fact twice in the inbox degrades signal-to-noise more than missing one finding for one tick.

Close out

One paragraph: which flows you scored, which you deferred below the volume floor and until when, what you added, which reports you authored or edited, what you ruled out and why. The harness saves this as the run summary; future runs read it via scout-runs-list. Do not write a separate "run metadata" scratchpad entry. "Scored the due flows, all conversions within baseline" is a real outcome.

Disqualifiers (skip these)

  • Denominator collapsed too. If the entrants/cohort size dropped alongside the rate, the flow isn't converting worse — fewer people entered. That's a capture or upstream-volume issue (→ anomaly-detection for the volume drop, session-replay/error-tracking if capture broke). Note it, hand off, don't file it as a conversion regression.
  • A running experiment explains it. If a live experiment targets the flow's flag, a conversion shift in the exposed population is the experiment doing its job. Check product_intents / running experiments; only author if the move is outside the experiment's exposed users or the experiment can't account for the magnitude. Experiment validity is the experiments scout's job, not yours.
  • Flow-definition change, not behavior. If someone edited the funnel's steps, the retention event, or the date range, the rate "moved" because the measurement did. Read the insight's recent last_modified_at and query JSON before trusting a delta.
  • Seasonal swings — weekday/weekend, business-hours rhythm, end-of-month. Real only once the move clears the seasonality-matched baseline (compare same-weekday windows).
  • The current partial window — never score the in-progress day/week.
  • An immature retention cell — never score a retention cell whose return interval has not ended at the observation cutoff, even when its acquisition cohort is complete.
  • An immature funnel window — never score a funnel entrant window whose final entrant has not had the full saved conversion interval, plus the ingestion buffer, to convert, even when the window is calendar-complete.
  • Low-volume flows — funnels/cohorts whose entrant counts are too small for a stable rate (enforce a minimum-entrants floor; a few users' movement is not signal). Never report on them, and defer their next check with the volume gate instead of scoring them every run.
  • Single known internal/test cohort — a conversion change driven only by internal distinct_ids or a dev/test environment segment.
  • Known launches / migrations / backfills the team already knows about — if a noise: / addressed: entry names it, skip.

When in doubt, refresh the baseline memory instead of filing a report. A false conversion-regression alarm erodes trust fast.

MCP tools

Direct (read-only):

  • query-funnel — score a funnel's step-to-step conversion over a window (the primary scorer for funnel flows; re-run per prior window for the baseline, and with a breakdown to attribute).
  • query-retention — cohort return rates per day-offset (retention cliffs).
  • query-lifecycle / query-stickiness — composition + engagement-frequency shifts.
  • query-paths — infer the dominant activation sequence when seeding an inferred flow.
  • query-trends — sanity-check the entrant denominator volume behind a rate.
  • insight-get — read a saved flow's steps/events/filters before scoring.
  • insights-list / execute-sql over system.insights — find saved funnel/retention/ lifecycle/stickiness insights (query::text ILIKE '%FunnelsQuery%' etc.) and their recency.
  • read-data-schema — confirm events/properties before any SQL or inferred funnel.
  • inbox-reports-list / inbox-reports-retrieve — the reports already in the inbox; check before authoring so you edit instead of duplicating (ordering=-updated_at).
  • inbox-report-artefacts-list — a comparable report's artefact log, where the routed suggested_reviewers live (the report record doesn't expose them) — reviewer precedent.
  • scout-members-list — this project's members with their resolved github_login, to route suggested_reviewers to a flow / product-area owner. The in-run roster (the org-scoped resolver tools aren't available in a scout run) — but prefer routing by the flow's created_by user_uuid (resolved server-side) when your evidence already names it.

Harness-level:

  • scout-project-profile-get / scout-scratchpad-search / scout-runs-list / scout-runs-retrieve — orientation + dedupe.
  • scout-emit-report / scout-edit-report / scout-scratchpad-remember / scout-scratchpad-forget — author a report / edit an existing one / remember.

When to stop

  • No flow worth watching, or every flow deferred below the volume floor (quick close-out) → close out empty.
  • You've scored the due watchlist flows and added a couple of new ones → close out, even if more remain. Each run advances the watchlist.
  • A candidate matches a noise: / addressed: / dedupe: entry, or an existing inbox report → edit-or-skip.

Fewer, well-calibrated, denominator-checked regressions beat a flood of seasonal or volume-driven false positives.