SKILL.md · 24000
st by the specific experiment name **and** flag key (ordering=-updated_at), not a broad word like experiment` (which matches hundreds of unrelated UX reports).
- Edit (
scout-edit-report) when a still-live report already covers the same experiment issue — an SRM still skewed, a stall that hasn't recovered, a$multipletrend still climbing. Add the fresh numbers withappend_evidence, or rewrite the title/summary on a report you authored. This is the default when a match exists.edit-reportcan't change status, so if the matched report isresolved/suppressed/failed, don't append (it won't resurface) — author a fresh report for the relapse and repoint thereport:key. - Author (
scout-emit-report) only when nothing live covers it. A good report names the experiment id and flag key, quantifies the contradiction (observed vs expected split with exposure counts,$multiplepercentage, days dormant), passes the sample-size gate, and dates the onset — ideally tied to a flag version or activity-log entry. When the threat lives in a series — per-variant exposures skewing,$multipleshare climbing, a stall's flatline — attach it viachartsso the contradiction is visible, not just quoted. Setpriority(P0–P4) +priority_explanation— validity threats on a live decision (SRM, mid-run mutation, contamination) are P2, stalls P2–P3 by blast radius, lifecycle hygiene P3; it's the report's importance in the inbox, your call to make. Setsuggested_reviewersviascout-members-list(objects — a{github_login}or{user_uuid}, not bare strings; cache underreviewer:experiments:<slug>); left empty the report reaches no one. A validity threat is an investigation with a starting point: the flag, the assignment path, and the exposure data are code an agent can read, so setactionability=immediately_actionableand name therepositorythat owns the assignment code (omit the field when you can't tell which repo that is, so selection can find it). Keepactionability=requires_human_inputandrepository=NO_REPOfor a threat whose next step is the experiment owner's call — stop it, restart it, or discount the result (NO_REPO is what stopspriority+reviewers from spawning a pointless repo-selection sandbox). After authoring, write thereport:experiments:<slug>pointer with thereport_idso the next run edits instead of duplicating. - Remember if below the bar but worth carrying forward (a ratio drifting but inside the noise band,
$multiplecreeping at 0.3%, a plateau that needs one more week); skip with a one-line note if anoise:/addressed:/dedupe:entry or an existing report already covers it.
Sibling scouts share memory — the feature-flags scout owns non-experiment flag wiring, and the generalist (which ran an experiment-integrity lens before this specialist existed) may hold dedupe:general:experiment-* scratchpad entries; honor them like your own. When a prior run already covered a topic, default to edit-or-skip: the same fact twice in the inbox costs more than missing one finding for one tick.
Close out
Summarize the run in one paragraph: which experiments you checked, which reports you authored or edited, what you remembered, and what you ruled out. The harness saves it as the run summary; future runs read it via scout-runs-list. Don't write a separate "run metadata" scratchpad entry. "All running experiments healthy" is a real, useful outcome.
Disqualifiers (skip these)
- Launched < 24h ago — exposure precomputation lags ~15 min and day-one volume is unrepresentative; zero or skewed exposures right after launch are not findings yet.
- Ratio claims below the sample-size gate — no SRM call under ~1,000 bucketed exposures, and never inside the 3σ band. Low-volume splits wobble; that's variance.
- Metric movement — a variant winning, losing, or wobbling is the team's decision surface, not a scout finding. Only flag metric machinery (validity), with one exception: a long-stable answer on a zombie feeds the "decide now" recommendation.
- Paused experiments with no fresh exposures — that's what pause means. Check flag
activebefore calling a stall. - Rollout increases mid-run — the safe change; new users enter cleanly.
- Forced-variant release conditions (
filters.groups[].variantset) — deliberate non-random assignment; aggregate ratios won't match the nominal split by design. Note it once innoise:memory. - Declared A/A, placebo, or engine-validation experiments (name/description says A/A, placebo, validation, identical variants) — long runtimes and null results are the point; skip lifecycle "decide now" nudges. SRM checks still fully apply — a skewed A/A is exactly the kind of machinery fault these exist to catch. Note the intent once in
noise:memory. - Holdout-enrolled experiments — the holdout slice shifts effective ratios; read
holdout_idbefore judging a split. - Bucketing failures (
$feature_flag_response= false/empty) counted as variants — exclude from ratios; only their share trending up is interesting. - Experiments already concluded with a conclusion set — the team decided; lingering flag state is the only thing left worth checking.
When in doubt, write a memory entry instead of filing a report.
MCP tools
Direct calls (read-only):
experiment-list— cheap candidate discovery: id, name, status (draft / running / paused / stopped), dates,feature_flag_key. Filter bystatus; start here.experiment-results-get— the flagship detector: exposure block (total_exposures, dailytimeseries, nativesample_ratio_mismatch.p_value,bias_risk.multiple_variant_percentage) plus per-metricvalidation_failures/data: null. Heavy response with many metrics — read the exposure + validation fields, skip the per-metric stats. New-engine experiments only; passrefresh: false.experiment-get— full config for a candidate:running_time_calculation.recommended_sample_size,excluded_variants,exposure_criteria(customexposure_config.event,multiple_variant_handling,filterTestAccounts),resolved_exposure_event(the default exposure event to query),stats_config.method,holdout_id, linkedfeature_flag(active,version,bucketing_identifier,ensure_experience_continuity,filters.multivariate.variantsfor the configured split,filters.groups[].rollout_percentage,filters.groups[].variantoverrides),metrics(each withuuid+ fingerprint). Large response — candidates only.experiment-stats— project-wide velocity aggregate (launched / completed last 30d, active count). Cheap context for the hygiene pass.experiment-timeseries-results— day-by-day per-variant results for one metric (metric_uuid+fingerprintfrom the metrics array). Use sparingly, for the zombie "decide now" check.feature-flag-get-definition/feature-flags-activity-retrieve— flag state and edit-history diffs; the latter is how you date mid-run mutations.advanced-activity-logs-list(scopes: ["Experiment"]) — experiment-level edit timeline.execute-sqlagainstevents— exposure analysis. Properties:$feature_flag(flag key) +$feature_flag_response(variant, incl.$multiple) on the default exposure events ($feature_flag_called/$experiment_exposure);$feature/<flag-key>on custom exposure events.read-data-schema— confirm a custom exposure event and its properties exist before aggregating over them.
Inbox & reviewer routing:
inbox-reports-list/inbox-reports-retrieve— the reports already in the inbox; check before authoring so you edit instead of duplicating (ordering=-updated_at).inbox-report-artefacts-list— a comparable report's artefact log, where the routedsuggested_reviewerslive (the report record doesn't expose them) — reviewer precedent.scout-members-list— this project's members with theiruser_uuidand resolvedgithub_login. Route the owner with either value. Useuser_uuidwhengithub_loginis null. The org-scoped resolver tools are not available in a scout run.
Harness-level:
scout-project-profile-get/scout-scratchpad-search/scout-runs-list/scout-runs-retrieve— orientation + dedupe.scout-emit-report/scout-edit-report— author a report / edit an existing one (the report-channel contract is in the harness prompt).scout-scratchpad-remember/scout-scratchpad-forget— remember / prune stale memory keys.
When to stop
- No experiments in use →
not-in-use:entry, close out empty. - All running experiments match their config (ratio in band, fresh exposures, no post-launch flag edits) → close out empty; refresh
pattern:baselines if stale. - Candidates all gated by
noise:/addressed:/dedupe:entries, or an existing inbox report → edit-or-skip and close out. - You've filed (or edited) reports for what's solid → close out. One sharp validity report beats a laundry list of P3 hygiene nits.
"Looked but found nothing meaningful" is a real outcome.