SKILL.md · 24000
entries the previous page actually returned (not by limit), until a page comes back empty, then concatenate the pages before rendering; a short page is not the end, since a page also closes early at a byte cap, and stepping by limit past one would skip the entries it left out, so a single default page can drop the failure tail.
⚠️ Do not reach for exclude_types: "tool_call_update,…" to slim it down. It is tempting — the stream is dominated by incremental tool_call_update chunks — but each tool's actual input lives only in those chunks: the base tool_call event carries an empty rawInput, and the streamed updates build the input (and the final rawOutput) token by token.
Excluding them leaves you with tool names but no idea what the scout actually queried.
Fetch the full log and let the script reassemble each call (it groups by toolCallId, keeps the richest rawInput, and attaches the completion's rawOutput/status).
Whether a run wrote anything is a first-class field: emitted_report_ids / edited_report_ids. A non-empty emitted_report_ids lists the reports the run authored via emit_report, in order; edited_report_ids lists the reports it mutated via edit_report (which can target any inbox report, not just ones a scout authored).
A productive run typically has one id there and a summary like Report authored: <id>; resolve any id via inbox-reports-retrieve to read the report itself.
Don't parse the prose summary for output — a phrase like "already reported P1 … did not re-file" describes a prior run, so substring-matching the summary is unreliable; the id columns are the authoritative tally.
One scout shape breaks the equivalence between "no report ids" and "wrote nothing": a measurement scout files no report on a normal run because its output is the record stream, so check metadata.derived.has_structured_output (and the events themselves) before calling such a run empty.
Legacy runs: emitted_count / emitted_finding_ids. Runs from the deprecated signal-emitting channel (a scout without the allowed_tools opt-in — an old custom scout, or a canonical scout not yet ported) tally their output as emitted_count weak findings instead; each finding_id maps to a Signal with source_id = run:<run_id>:finding:<finding_id>.
For those runs only, scout-runs-emission-reports (pass the run_id) maps each emitted finding to the inbox report its signal grouped into (or null if it never surfaced).
On report-channel scouts these fields are always 0 / empty — don't diagnose off them.
See references/scout-data-model.md for the full field reference.
A run with status complete and an empty-handed summary ("surface at baseline, nothing to report") is a healthy outcome, not a failure — most runs should close out empty.
Treat a stream of empty close-outs as the fleet doing its job, not as the fleet being broken.
Workflow: inspect what the fleet has learned
The scratchpad is the fleet's durable, per-team memory — prose entries scouts write so future runs get smarter and quieter. Reading it tells you what the fleet believes about this project.
scout-scratchpad-search
{ "text": "error_tracking" }
Returns entries newest-first: 20 by default, limit up to 1000; text matches content and key case-insensitively.
Omit text to browse everything; pass keys_only=true for a cheap scan of what memories exist, or content_max_chars to preview long bodies, and date_to set to the oldest entry's updated_at to walk past the cap.
Expired entries (an expires_at in the past) are excluded unless include_expired=true; every entry also carries created_by_skill, the original creator (a scout's skill name, or pipeline:report-research / pipeline:implementation for a pipeline stage). It is lineage, not authorship of the current text: an upsert by another writer on the same team-wide key keeps both the original creator and created_by_run_id, so neither field names the current writer of a shared key; treat it as unknown rather than attributing the content to that skill.
Each entry's key carries a category prefix that tells you what kind of learning it is:
| Prefix | Meaning |
| ------------- | ---------------------------------------------------------------------------------------- |
| pattern: | A baseline: how this team's data normally shapes |
| watch: | A live issue being tracked but still below the report bar |
| followup: | A scout's validation queue: a probe a later run re-runs to confirm a fix or a recurrence |
| noise: | A pattern the fleet has decided to ignore (dev-only, single-user…) |
| addressed: | Something the team fixed or moved on from |
| dedupe: | A gate on re-filing a specific issue / fingerprint |
| allowlist: | Vetted entities never to re-surface |
| not-in-use: | A product/surface this team doesn't use (close-out memo) |
| mcp-gap: | A tooling gap a scout noticed worth raising later |
| improve: | A custom scout's suggested change to its own skill body, awaiting owner review |
| reported: | A canonical scout's record of a skill gap already fed back upstream to PostHog |
| report: | A report a scout authored: stores the report_id so later runs edit/dedup against it |
| reviewer: | A resolved owner (GitHub login) for an area, cached for suggested_reviewers routing |
This is the common vocabulary, not a closed set — scouts coin their own prefixes and <domain> labels as needed (the live fleet uses watch: heavily, for example), so treat an unfamiliar prefix as just another category.
Entries cross-reference each other with [[key]] wikilinks.
Keys follow <prefix>:<domain>:<entity> (e.g. dedupe:error_tracking:019e8375-…).
When a user asks "why isn't my scout flagging X anymore?", search the scratchpad for noise:, addressed:, dedupe:, and allowlist: entries — the fleet may have deliberately learned to suppress it.
The canonical prefix vocabulary and the four-state dedupe classifier the fleet reasons in terms of are documented in the authoring-scouts skill (references/dedupe-and-memory.md).
Custom scouts self-report skill improvements. A custom (team-authored) scout is invited by the harness to write an improve:<skill-name>:<topic> entry when a run produces concrete evidence its own skill body steered it wrong — the suggested change, the evidence, and a dated observed line, re-confirmed in place on later runs.
A custom scout on the report channel escalates recurring or material suggestions further: it files an inbox report about itself (titled Scout self-improvement: <skill-name> – <topic>) and stashes the report_id in the improve: entry — so the strongest suggestions reach the owner through the inbox like any other report, not only via the scratchpad.
When assessing a custom scout, search {"text": "improve:"} and surface these to the user: an entry re-confirmed across several runs is the highest-signal edit the owner can make.
Reviewing and applying them is a write operation — hand off to the authoring-scouts skill.
Canonical scouts never write improve: entries (their skill bodies are synced from PostHog's fleet), so an improve: entry under a canonical scout's domain is itself worth flagging.
Instead, a canonical scout routes skill-content gaps upstream to the PostHog team via the agent-feedback MCP tool (feedback_type: "scout"), generalized so no project data travels, and keeps a reported:<skill-name>:<topic> entry as its local record of what it already submitted — so a reported: entry tells you a gap has been raised with PostHog, not with this team.
Workflow: see what scouts have written
Scout output reaches the user as inbox reports. Filter the inbox to one scout by its skill name (comma-separate several):
inbox-reports-list
{ "scout": "signals-scout-error-tracking", "limit": 20 }
Each row also carries scout_name, so a fleet-wide page tells you which scout authored what.
For the whole fleet at once, filter on the source instead: { "source_product": "signals_scout", "limit": 20 }.
Every report a scout authors carries backing signals tagged source_product="signals_scout", and that filter keeps any report whose contributing signals include the tag, so the result is the set of reports the fleet has authored.
Neither filter captures edit-only work: a scout that edits an existing non-scout report (appending a note to a pipeline report, say) adds no signals_scout signal, so that report won't match; trace edits through the run rows' edited_report_ids instead.
Suppressed reports are hidden by default, so add include_all_statuses: true (or status: "suppressed") when judging how much of a scout's output got filtered. Even then both the scout and the source_product filter miss a report the safety judge suppressed, because each resolves reports through their signals and an unsafe report's signals are never indexed; the run rows' emitted_report_ids are the complete enumeration for a signal-to-noise audit.
An empty result means the fleet hasn't authored any reports (yet), not that the filter is broken.
Scouts hold a high bar — most runs close out without writing — so on a quiet or newly enrolled project zero scout reports is the normal, expected state.
Note the inbox only shows surfaced reports: a report the safety judge suppressed (or one filed as not_actionable) persists with status SUPPRESSED but doesn't appear in the default inbox view.
For the per-run view, work from the runs instead: scout-runs-list?emitted=true lists every run that authored a report, and each run's emitted_report_ids / edited_report_ids name exactly which reports it wrote or updated — resolve them via inbox-reports-retrieve.
The flip side matters when explaining a gap: a run can narrate "authored a report" in its summary yet have the write silently dropped by a preflight gate (dry-run at the time, the org hasn't approved AI processing, or the signals_scout source is disabled) — those leave emitted_report_ids empty, so a claimed-but-absent report is itself a diagnostic.
To browse the inbox more broadly, use the inbox-exploration skill (statuses, suggested reviewers, drilling into a report's underlying signals).
The report contract behind each report — the report bar, evidence, actionability, reviewer routing — is documented in the authoring-scouts skill (references/report-contract.md).
A report's status is not its whole state: read its checks too. A check is a follow-up measurement a scout or the report pipeline attached to a report — an expectation plus a time to test it — so a report can be resolved and still under measurement.
inbox-report-checks-list returns every check on one report, newest first, with its status (pending, active, passed, failed, errored, expired, cancelled), its last_outcome, and its schedule (next_run_at, run_interval_minutes, runs_remaining, expires_at); inbox-report-checks-retrieve returns one check with its full config.
Read the rows this way:
pending— the check is waiting for the report to resolve before its clock starts.activewith a futurenext_run_at— a verdict is on its way. Don't re-derive the answer by hand; say when it lands.- Everything else is terminal. A claim still worth watching needs a new check, which is a write — hand off to
authoring-scouts.
The verdicts themselves are not on the check row: each one is a check_result artefact on the report, so read them through the report's artefact list.
query and baseline_value read as null for a credential that cannot read the data they describe.
A failed check on a resolved report usually has a fresh report behind it as well (the breach is re-surfaced as a new report linked to the resolved one), so look for that before reporting the relapse as unhandled.
The full mechanics — the two check kinds, the soak window, what each verdict does next — are in the authoring-scouts skill (references/report-checks.md).
Workflow: assess health and performance
"Is my scout actually working / earning its cost?"
There's no single metric — judge a scout over a window of runs.
Pull the runs (scout-runs-list with a date_from), then reason across the dimensions below.
The full playbook, including how to read each signal and the common failure modes, is in references/assessing-performance.md.
- Cadence adherence: are runs landing roughly every
run_interval_minutes, or on eachrun_cron_scheduleslot? Judge a cron scout against its slots, not its interval: a weekday-only scout's weekend gap is the schedule, not a stall, andassess_health.pyskips interval-based scoring for it (its cadence column readscron). Large gaps mean the coordinator is skipping it (disabled, drained from the flag, or capped out on busy ticks) — or it's dispatching but the runs aren't materializing. Tell the two apart withlast_run_at: if the config'slast_run_atis also stale, the coordinator stopped planning it; iflast_run_atis fresh but the newest run row is hours old, it's the dispatch-vs-execution divergence above (workers backed up / down, or runs stranded), whichscout-runs-listalone hides. - Success rate — how many runs reach a clean
statusvs. error out? A run of errors is a broken scout, not a quiet one; group the failed runs byfailure_reasonto see whether it is one cause repeating. - Report rate — what fraction of runs wrote or edited a report vs. closed out empty.
Read it straight off
emitted_report_ids/edited_report_idsper run (or split the window withruns-list?emitted=true/?emitted=false, remembering edit-only runs read as not-emitted). Near-zero over a long window on a live surface can mean the discriminator is too strict (or the surface really is quiet); near-100% usually means it's too noisy. Don't score a measurement scout on this at all — a near-zero report rate is its designed behavior, so read its record stream instead. Most healthy scouts write rarely. - Signal-to-noise — of what it wrote, how much surfaced as actionable inbox reports vs. got suppressed or dismissed?
Resolve each run's
emitted_report_idsviainbox-reports-retrieveand read the report statuses — across a window, the share of authored reports that are live and non-suppressed is the scout's hit rate. - Memory growth — a healthy scout accumulates
pattern:/noise:/dedupe:entries over time. A scout with an empty scratchpad after many runs isn't learning.
Helper scripts
The skill bundles three pure formatters under scripts/ for the most common asks.
They do no network I/O — they are the back half of an "agent fetches, script formats" split.
The pattern is always the same:
- Fetch each payload with the MCP using
call --json(raw JSON, not the pretty text format) and save it to a file. For the big ones (scout-runs-list,tasks-runs-session-logs-retrieve) this is mandatory anyway — they overflow inline and spill to a file you can point the script at. - Run the script over those files.
All three are stdlib-only Python 3.11+ and print plain text to stdout (or --out) — designed to read well in a terminal, so save them as .txt.
scripts/render_run_report.py — drill into one run
Produces the kind of detailed write-up you'd want when inspecting a single run: header (status, duration, posture), a narrated timeline that interleaves the agent's narration with each tool call and its real input, the end-of-run summary, and any scratchpad memory.
# fetch (note --json), saving each to a file:
# call --json scout-runs-retrieve { "id": "<run_id>" } -> run.json
# call --json tasks-runs-session-logs-retrieve { "id": "<task_run_id>", "task_id": "<task_id>", "limit": 5000, "offset": 0 } -> log.json (FULL, no exclude_types; page with offset until an empty page, then concatenate)
# (optional) call --json scout-scratchpad-search { ... } -> mem.json
# (optional) call --json scout-config-list {} -> cfg.json
python scripts/render_run_report.py --run run.json --log log.json \
--scratchpad mem.json --config cfg.json --out report.txt
Modes (--mode, default detailed):
| Mode | Contains | --log needed? |
| ---------- | ------------------------------------------------------------------ | --------------- |
| summary | header + posture + close-out prose | no |
| detailed | + narrated timeline with tool inputs + tool tally + scratchpad | yes |
| full | + each tool call's (truncated) output inline | yes |
Other flags: --show-output (outputs in detailed mode), --input-width / --output-width (truncation), --no-art (skip the hedgehog banner), --base-url (defaults to us.posthog.com).
scripts/fleet_survey.py — survey the whole fleet
One scannable table — scout, enabled, posture, cadence, last run, last outcome — with a "worth a look" section that flags never-run, stuck-in-dry-run, and last-run-failed scouts.
# call --json scout-config-list {} -> cfg.json
# (optional) call --json scout-runs-list { "limit": 30 } -> runs.json (small limit!)
python scripts/fleet_survey.py --config cfg.json --runs runs.json --now <current-ISO-time>
Pass --now (the current time, ISO-8601) to get relative "ago" columns; the last-outcome column reads what the run wrote straight off emitted_report_ids / edited_report_ids on the run row.
scripts/assess_health.py — health over a window of runs
Implements the "assess health and performance" workflow above: a per-scout table (runs, success %, report rate, cadence gap vs interval, adherence, median duration, memory growth) plus a "worth a look" section flagging all-failed scouts, timeout-shaped failures, cadence stalls, staleness, and empty scratchpads.
# call --json scout-runs-list { "limit": 100, "date_from": "<ISO>" } -> runs.json
# (optional) call --json scout-config-list {} -> cfg.json
# (optional) call --json scout-scratchpad-search {} -> mem.json
python scripts/assess_health.py --runs runs.json --config cfg.json \
--scratchpad mem.json --now <current-ISO-time> [--skill signals-scout-general]
--config is what lets it score cadence adherence (the expected interval; a scout with a run_cron_schedule is marked cron and exempt from the interval-based adherence, stall, and staleness flags) and staleness (the authoritative last_run_at, which the windowed runs can miss when the 100-row cap truncates the newest runs).
Without --scratchpad the memory column shows n/a and no memory flags fire.
The report rate reads the run rows' emitted_report_ids / edited_report_ids directly, so it's exact — but it only counts writes; judge signal-to-noise by the resulting report statuses via inbox-reports-list.
Tips
- Always surface the
emitposture. "Running but in dry-run" is the single most common reason a user thinks a scout is broken when it isn't. - An empty close-out is success. Most runs should find nothing. Don't report a wall of clean, empty runs as a problem.
- What a run wrote is a first-class run field. Read
emitted_report_ids/edited_report_idsper run (or filter withruns-list?emitted=true) to find what was written, without parsing the prosesummary. Thesource_product: "signals_scout"inbox filter lists the reports the fleet surfaced; an empty result there means it hasn't written anything yet (scouts hold a high bar), not that the filter is broken. emitted_count: 0does not mean "did nothing".emitted_count/emitted_finding_idsare legacy signal-channel fields — they stay0/ empty on report-channel scouts, productive or not. Judge output by the report columns; a non-zero legacy tally means the run came from a scout still on the legacy channel, and is that run's real output.- A
failedrun that lasted ~15 minutes is usually a timeout, not a broken scout. The per-run budget is 15 minutes; completed runs finish in a couple.failure_reasonnames a timeout outright, so check it before reasoning from duration. Most often the scout over-investigated and ran the full budget (the fleet self-corrects by writing "tight-run recipe" scratchpad entries) — but some are false timeouts where the scout actually finished in a few minutes and the run then hung on a dropped close-out. The session log (above) tells them apart: real over-investigation shows tool calls right up to the wall; a false timeout goes silent long before it. Don't assume over-investigation from duration alone. - Lead with the run
summary, then offertask_urlfor the full transcript — don't dump raw run rows at the user. last_run_at: nullmeans the coordinator has never dispatched the scout, not that it has never run: manual (scout-run-now) and workflow-triggered runs leave the stamp untouched, so checkscout-runs-listbefore calling a scout never-run. Then check it's enabled and the project is enrolled (a cron scout also simply waits for its first slot after creation).- To explain a quiet scout, read the project profile.
scout-project-profile-getshows whether the surface it watches is even in use — a logs scout on a project with no logs has nothing to do. - This skill is read-only. To change a scout's schedule, posture, or body, hand off to the
authoring-scoutsskill — it coversscout-config-updateand the skills-store edit path.