SKILL.md · 24000
eport is no longer live, author a fresh report for the relapse and repoint that rate's report:product_analytics:flow:<short_id>:<rate> entry at the new id.
- Author a fresh report via
scout-emit-reportwhen nothing in the inbox covers it (or a known regression has new evidence that changes the verdict). A strong finding here: the rate dropped clearly below the flow's seasonality-matched baseline (robust z ≥ ~3, or a conversion-point drop beyond the baseline band), the entrant denominator held (quantify both — "step-2 conversion 62%→48% while step-1 entrants steady at ~5.2k/day"), the move is broad across segments (not one known cohort), and it's not explained by a running experiment or a flow-definition edit. Put the flowshort_id, the latest-window rate, the baseline band, the per-step/per-cohort numbers, the entrant volumes, and the time window in theevidence. A rate regression is chart-shaped — attach the regressed rate viacharts, matched to the flow type: a funnel or retention rate with its steady entrant denominator alongside (the drop-on-steady-entrants shape), lifecycle as the state composition over time, stickiness as the frequency distribution against its population. A behavioral regression is an investigation with a starting point, not a question for a person: the flow, the step that moved, and the onset are where an agent begins, so setactionability=immediately_actionable, setpriority, and name therepositorythat owns the flow (omit the field when you can't tell which repo that is, so selection can find it). Keepactionability=requires_human_inputfor a move whose next step is a product call — whether the new behavior was intended, which of several valid flows should win — and on those leavepriorityandrepositoryunset, since they're PR-autostart fields and supplyingpriority+suggested_reviewerswith norepositorysignals PR intent that spins up a repo-selection sandbox only to no-op (autostart needsimmediately_actionable). Reach for them (P2 broad regression on a human-saved flow, P3 single-segment /inferred) only on the rare regression you'd actually want a draft PR for. Setsuggested_reviewerswhenever you can confidently resolve one — each entry is{github_login?, user_uuid?}, and the usual route here is to pass the flow's owning person as auser_uuid, or reuse a cachedreviewer:product_analytics:<area>login.user_uuidresolution is fail-loud: acreated_bythat is not an org member rejects the wholeemit-report, not only the reviewer. Do not pass a rawcreated_byif you cannot verify membership. Prefer a cached login or acreated_bythat routed before. You can also author the report without a reviewer and add one later.scout-members-listgives this project's members with theiruser_uuidand resolvedgithub_login. Routing helps the report reach a human, so resolve an owner when you can. After authoring, write a rate-scopedreport:product_analytics:flow:<short_id>:<rate>scratchpad entry with thereport_id. Use a separate entry for each rate. The harness prompt carries the full report-channel contract. - Remember if suggestive but below that bar, or to refresh a baseline.
- Skip if a
noise:/addressed:/dedupe:entry, or an existing inbox report, already covers it.
If anomaly-detection already owns a related metric move in the inbox, author only if your behavioral-rate angle is materially new; otherwise edit-or-skip. The same fact twice in the inbox degrades signal-to-noise more than missing one finding for one tick.
Close out
One paragraph: which flows you scored, which you deferred below the volume floor and until when, what you added, which reports you authored or edited, what you ruled out and why. The harness saves this as the run summary; future runs read it via scout-runs-list. Do not write a separate "run metadata" scratchpad entry. "Scored the due flows, all conversions within baseline" is a real outcome.
Disqualifiers (skip these)
- Denominator collapsed too. If the entrants/cohort size dropped alongside the rate, the flow isn't converting worse — fewer people entered. That's a capture or upstream-volume issue (→
anomaly-detectionfor the volume drop,session-replay/error-trackingif capture broke). Note it, hand off, don't file it as a conversion regression. - A running experiment explains it. If a live experiment targets the flow's flag, a conversion shift in the exposed population is the experiment doing its job. Check
product_intents/ running experiments; only author if the move is outside the experiment's exposed users or the experiment can't account for the magnitude. Experiment validity is theexperimentsscout's job, not yours. - Flow-definition change, not behavior. If someone edited the funnel's steps, the retention event, or the date range, the rate "moved" because the measurement did. Read the insight's recent
last_modified_atand query JSON before trusting a delta. - Seasonal swings — weekday/weekend, business-hours rhythm, end-of-month. Real only once the move clears the seasonality-matched baseline (compare same-weekday windows).
- The current partial window — never score the in-progress day/week.
- An immature retention cell — never score a retention cell whose return interval has not ended at the observation cutoff, even when its acquisition cohort is complete.
- An immature funnel window — never score a funnel entrant window whose final entrant has not had the full saved conversion interval, plus the ingestion buffer, to convert, even when the window is calendar-complete.
- Low-volume flows — funnels/cohorts whose entrant counts are too small for a stable rate (enforce a minimum-entrants floor; a few users' movement is not signal). Never report on them, and defer their next check with the volume gate instead of scoring them every run.
- Single known internal/test cohort — a conversion change driven only by internal distinct_ids or a
dev/testenvironment segment. - Known launches / migrations / backfills the team already knows about — if a
noise:/addressed:entry names it, skip.
When in doubt, refresh the baseline memory instead of filing a report. A false conversion-regression alarm erodes trust fast.
MCP tools
Direct (read-only):
query-funnel— score a funnel's step-to-step conversion over a window (the primary scorer for funnel flows; re-run per prior window for the baseline, and with a breakdown to attribute).query-retention— cohort return rates per day-offset (retention cliffs).query-lifecycle/query-stickiness— composition + engagement-frequency shifts.query-paths— infer the dominant activation sequence when seeding an inferred flow.query-trends— sanity-check the entrant denominator volume behind a rate.insight-get— read a saved flow's steps/events/filters before scoring.insights-list/execute-sqloversystem.insights— find saved funnel/retention/ lifecycle/stickiness insights (query::text ILIKE '%FunnelsQuery%'etc.) and their recency.read-data-schema— confirm events/properties before any SQL or inferred funnel.inbox-reports-list/inbox-reports-retrieve— the reports already in the inbox; check before authoring so you edit instead of duplicating (ordering=-updated_at).inbox-report-artefacts-list— a comparable report's artefact log, where the routedsuggested_reviewerslive (the report record doesn't expose them) — reviewer precedent.scout-members-list— this project's members with their resolvedgithub_login, to routesuggested_reviewersto a flow / product-area owner. The in-run roster (the org-scoped resolver tools aren't available in a scout run) — but prefer routing by the flow'screated_byuser_uuid(resolved server-side) when your evidence already names it.
Harness-level:
scout-project-profile-get/scout-scratchpad-search/scout-runs-list/scout-runs-retrieve— orientation + dedupe.scout-emit-report/scout-edit-report/scout-scratchpad-remember/scout-scratchpad-forget— author a report / edit an existing one / remember.
When to stop
- No flow worth watching, or every flow deferred below the volume floor (quick close-out) → close out empty.
- You've scored the due watchlist flows and added a couple of new ones → close out, even if more remain. Each run advances the watchlist.
- A candidate matches a
noise:/addressed:/dedupe:entry, or an existing inbox report → edit-or-skip.
Fewer, well-calibrated, denominator-checked regressions beat a flood of seasonal or volume-driven false positives.