AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

signals-scout-experiments

Signals scout for PostHog A/B experiments. Watches running experiments for validity threats — sample ratio mismatch, contamination, exposure stalls, mid-run flag mutations — and lifecycle drift.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--signals-scout-experiments at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 1

SKILL.md · 24000

st by the specific experiment name **and** flag key (ordering=-updated_at), not a broad word like experiment` (which matches hundreds of unrelated UX reports).

  • Edit (scout-edit-report) when a still-live report already covers the same experiment issue — an SRM still skewed, a stall that hasn't recovered, a $multiple trend still climbing. Add the fresh numbers with append_evidence, or rewrite the title/summary on a report you authored. This is the default when a match exists. edit-report can't change status, so if the matched report is resolved / suppressed / failed, don't append (it won't resurface) — author a fresh report for the relapse and repoint the report: key.
  • Author (scout-emit-report) only when nothing live covers it. A good report names the experiment id and flag key, quantifies the contradiction (observed vs expected split with exposure counts, $multiple percentage, days dormant), passes the sample-size gate, and dates the onset — ideally tied to a flag version or activity-log entry. When the threat lives in a series — per-variant exposures skewing, $multiple share climbing, a stall's flatline — attach it via charts so the contradiction is visible, not just quoted. Set priority (P0–P4) + priority_explanation — validity threats on a live decision (SRM, mid-run mutation, contamination) are P2, stalls P2–P3 by blast radius, lifecycle hygiene P3; it's the report's importance in the inbox, your call to make. Set suggested_reviewers via scout-members-list (objects — a {github_login} or {user_uuid}, not bare strings; cache under reviewer:experiments:<slug>); left empty the report reaches no one. A validity threat is an investigation with a starting point: the flag, the assignment path, and the exposure data are code an agent can read, so set actionability=immediately_actionable and name the repository that owns the assignment code (omit the field when you can't tell which repo that is, so selection can find it). Keep actionability=requires_human_input and repository=NO_REPO for a threat whose next step is the experiment owner's call — stop it, restart it, or discount the result (NO_REPO is what stops priority+reviewers from spawning a pointless repo-selection sandbox). After authoring, write the report:experiments:<slug> pointer with the report_id so the next run edits instead of duplicating.
  • Remember if below the bar but worth carrying forward (a ratio drifting but inside the noise band, $multiple creeping at 0.3%, a plateau that needs one more week); skip with a one-line note if a noise: / addressed: / dedupe: entry or an existing report already covers it.

Sibling scouts share memory — the feature-flags scout owns non-experiment flag wiring, and the generalist (which ran an experiment-integrity lens before this specialist existed) may hold dedupe:general:experiment-* scratchpad entries; honor them like your own. When a prior run already covered a topic, default to edit-or-skip: the same fact twice in the inbox costs more than missing one finding for one tick.

Close out

Summarize the run in one paragraph: which experiments you checked, which reports you authored or edited, what you remembered, and what you ruled out. The harness saves it as the run summary; future runs read it via scout-runs-list. Don't write a separate "run metadata" scratchpad entry. "All running experiments healthy" is a real, useful outcome.

Disqualifiers (skip these)

  • Launched < 24h ago — exposure precomputation lags ~15 min and day-one volume is unrepresentative; zero or skewed exposures right after launch are not findings yet.
  • Ratio claims below the sample-size gate — no SRM call under ~1,000 bucketed exposures, and never inside the 3σ band. Low-volume splits wobble; that's variance.
  • Metric movement — a variant winning, losing, or wobbling is the team's decision surface, not a scout finding. Only flag metric machinery (validity), with one exception: a long-stable answer on a zombie feeds the "decide now" recommendation.
  • Paused experiments with no fresh exposures — that's what pause means. Check flag active before calling a stall.
  • Rollout increases mid-run — the safe change; new users enter cleanly.
  • Forced-variant release conditions (filters.groups[].variant set) — deliberate non-random assignment; aggregate ratios won't match the nominal split by design. Note it once in noise: memory.
  • Declared A/A, placebo, or engine-validation experiments (name/description says A/A, placebo, validation, identical variants) — long runtimes and null results are the point; skip lifecycle "decide now" nudges. SRM checks still fully apply — a skewed A/A is exactly the kind of machinery fault these exist to catch. Note the intent once in noise: memory.
  • Holdout-enrolled experiments — the holdout slice shifts effective ratios; read holdout_id before judging a split.
  • Bucketing failures ($feature_flag_response = false/empty) counted as variants — exclude from ratios; only their share trending up is interesting.
  • Experiments already concluded with a conclusion set — the team decided; lingering flag state is the only thing left worth checking.

When in doubt, write a memory entry instead of filing a report.

MCP tools

Direct calls (read-only):

  • experiment-list — cheap candidate discovery: id, name, status (draft / running / paused / stopped), dates, feature_flag_key. Filter by status; start here.
  • experiment-results-get — the flagship detector: exposure block (total_exposures, daily timeseries, native sample_ratio_mismatch.p_value, bias_risk.multiple_variant_percentage) plus per-metric validation_failures / data: null. Heavy response with many metrics — read the exposure + validation fields, skip the per-metric stats. New-engine experiments only; pass refresh: false.
  • experiment-get — full config for a candidate: running_time_calculation.recommended_sample_size, excluded_variants, exposure_criteria (custom exposure_config.event, multiple_variant_handling, filterTestAccounts), resolved_exposure_event (the default exposure event to query), stats_config.method, holdout_id, linked feature_flag (active, version, bucketing_identifier, ensure_experience_continuity, filters.multivariate.variants for the configured split, filters.groups[].rollout_percentage, filters.groups[].variant overrides), metrics (each with uuid + fingerprint). Large response — candidates only.
  • experiment-stats — project-wide velocity aggregate (launched / completed last 30d, active count). Cheap context for the hygiene pass.
  • experiment-timeseries-results — day-by-day per-variant results for one metric (metric_uuid + fingerprint from the metrics array). Use sparingly, for the zombie "decide now" check.
  • feature-flag-get-definition / feature-flags-activity-retrieve — flag state and edit-history diffs; the latter is how you date mid-run mutations.
  • advanced-activity-logs-list (scopes: ["Experiment"]) — experiment-level edit timeline.
  • execute-sql against events — exposure analysis. Properties: $feature_flag (flag key) + $feature_flag_response (variant, incl. $multiple) on the default exposure events ($feature_flag_called / $experiment_exposure); $feature/<flag-key> on custom exposure events.
  • read-data-schema — confirm a custom exposure event and its properties exist before aggregating over them.

Inbox & reviewer routing:

  • inbox-reports-list / inbox-reports-retrieve — the reports already in the inbox; check before authoring so you edit instead of duplicating (ordering=-updated_at).
  • inbox-report-artefacts-list — a comparable report's artefact log, where the routed suggested_reviewers live (the report record doesn't expose them) — reviewer precedent.
  • scout-members-list — this project's members with their user_uuid and resolved github_login. Route the owner with either value. Use user_uuid when github_login is null. The org-scoped resolver tools are not available in a scout run.

Harness-level:

  • scout-project-profile-get / scout-scratchpad-search / scout-runs-list / scout-runs-retrieve — orientation + dedupe.
  • scout-emit-report / scout-edit-report — author a report / edit an existing one (the report-channel contract is in the harness prompt).
  • scout-scratchpad-remember / scout-scratchpad-forget — remember / prune stale memory keys.

When to stop

  • No experiments in use → not-in-use: entry, close out empty.
  • All running experiments match their config (ratio in band, fresh exposures, no post-launch flag edits) → close out empty; refresh pattern: baselines if stale.
  • Candidates all gated by noise: / addressed: / dedupe: entries, or an existing inbox report → edit-or-skip and close out.
  • You've filed (or edited) reports for what's solid → close out. One sharp validity report beats a laundry list of P3 hygiene nits.

"Looked but found nothing meaningful" is a real outcome.