references/scout-patterns.md
Scout patterns (a cookbook)
A catalog of the reference architectures scouts fall into.
Most new scouts are a variation on one of these — pick the closest shape as your starting point, copy the named canonical scout it maps to, and swap in your surface's discriminator and queries.
The scout-anatomy.md body structure is the same for all of them; what changes between patterns is what the scout watches, how it reads that data, and what its signal-vs-noise discriminator is.
This is a living reference — add a pattern when a genuinely new shape proves itself, rather than letting every scout reinvent one.
Contents
- What a scout can watch
- The patterns: anomaly watcher · liveness / absence watcher · zero-result / unmet demand · watchlist (explore/exploit + curated) · cross-product correlation · recommendation / gap · warehouse-backed source · custom / single-event · open-text theme · adversarial / abuse concentration · external-tool / code · state ∩ code-intersection · custom issue-tracker / work-queue · daily digest / roll-up · triage over a pre-detected stream · first-person dogfooding / probe · recurring measurement / LLM-judge · maintainer / steward · owner-scoped book / queue · trigger-to-brief enrichment · dispatcher / campaign · fleet meta-scout / reviewer
- Safety: treat ingested content as untrusted data
- Cross-cutting techniques
- Picking and combining
What a scout can watch
The single most useful thing to internalize: a scout is not limited to PostHog analytics events. It can watch anything the project can see, and the report / dedupe / memory contract is identical regardless of where the data comes from.
| Source | How the scout reads it |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Collected events | read-data-schema to confirm the event + properties, then query-* tools or execute-sql. The common case. |
| The data warehouse | execute-sql over system.information_schema.* to confirm columns, then execute-sql. Any source PostHog ingests becomes a queryable table — see the warehouse-backed pattern below. |
| PostHog product entities | dedicated list/get tools (insights, dashboards, surveys, error issues, experiments, flags) plus execute-sql over system.*. |
| External systems | from inside the sandbox — a CLI tool, a public git repo, an HTTP API. The default TRUSTED network covers the platform allowlist (GitHub, package registries); set network_access=full on the scout's config for anything outside it. See the external-tool pattern. |
| Other agents' output | Replay Vision observations ($recording_observed), PostHog AI conversations, the report pipeline's own verdicts and research notes, sibling scouts' reports and scratchpad. Ordinary events and tools, but a different kind of evidence: a judgment somebody else's model already made, which a scout can corroborate with, aggregate over, or audit. See the owner-scoped, judge-of-a-judge, and fleet meta-scout patterns. |
| The scout fleet itself | scout-config-list, scout-runs-list / -retrieve, scout-scratchpad-search, and inbox-reports-list filtered by scout. A scout can watch how the other scouts are doing. See the fleet meta-scout pattern. |
The warehouse row is the big unlock: once a Slack channel, a Stripe account, a CRM, a billing system, a support inbox, a social-listening feed, or an app database (via CDC) is synced into the warehouse, a scout queries it with execute-sql exactly like it queries events — and the watched surface need not be PostHog analytics at all.
The patterns
| Pattern | Watch this when… | Canonical example |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Anomaly watcher | a product surface has a metric with a baseline that can move (bursts, drops, regressions). | signals-scout-error-tracking, -logs, -revenue-analytics, -csp-violations |
| Liveness / absence watcher | the signal is an expected event not happening — a control gone silent, a promise unfulfilled, an automation stalled. | (see detailed patterns and variants below) |
| Zero-result / unmet demand | a request succeeds but comes back empty — the failure is in what was returned, not in whether it worked. | a search / catalog supply-gap scout (below) |
| Watchlist (explore/exploit, or curated) | the surface has more to watch than one run can cover — discovered over time (explore/exploit) or a fixed set you already know matters (curated). | signals-scout-anomaly-detection (discovered); a curated-dashboard scout (below) |
| Cross-product correlation | the question spans products — a cause in one surface, an effect in another. | signals-scout-general |
| Recommendation / gap | nothing is broken, but the team is missing coverage or following an anti-pattern. | signals-scout-observability-gaps |
| Warehouse-backed source | the signal lives in a non-PostHog source synced into the warehouse. | a Slack-channel-sync scout (below) |
| Custom / single-event | one bespoke event carries the whole signal. | an MCP-feedback scout (below) |
| Open-text theme | the data is free text and the value is in recurring themes, not individual rows. | signals-scout-surveys (open-text); brand/feedback scouts |
| Adversarial / abuse concentration | the watched party benefits from not being caught — incentive farming, scraping, spam, multi-accounting. | a trial-credit-farming scout (below) |
| External-tool / code | the judgement comes from running a tool or reading code, not from analytics. | a static-analysis CLI scout (below) |
| State ∩ code intersection | the signal is the overlap of a PostHog entity's state and what's in the source repo. | a feature-flag-cleanup scout (below) |
| Custom issue-tracker / work-queue | a built-in signals source (GitHub, Linear) already ingests the tracker, but you need scoping or judgment its config can't express. | a GitHub-issue readiness scout (below) |
| Daily digest / roll-up | the team wants a scheduled, human-readable synthesis of a surface — one report a day, quiet or not. | an AI-observability daily-digest scout (below) |
| Triage over a pre-detected stream | a detector already exists (spikes, alerts, health checks, a bot-run triage channel) and the job is judgment, not detection. | signals-scout-health-checks, -insight-alerts; a spike-triage scout (below) |
| First-person dogfooding / probe | the watched surface is something an agent can use, and the freshest signal is friction experienced first-hand. | an MCP-surface dogfooding scout (below) |
| Recurring measurement / LLM-judge | the deliverable is a data series, not a report — a recurring judgment, extraction, or snapshot no deterministic query can compute. | a content-quality judge scout (below) |
| Maintainer / steward | the scout owns a set of PostHog objects (dashboards, alerts, warehouse views, scanner prompts, a skill) and should fix what rots, not only describe the fix. | a dashboard steward, an alert-fleet steward, a warehouse-view steward (below) |
| Owner-scoped book / queue | one person's accounts, tickets, pull requests, or issues, watched for the few things that person should act on next. | an account-book scout, a personal PR sensor (below) |
| Trigger-to-brief enrichment | an upstream rule already names a new entity (a signup, an eligible account, a new arrival) and the value is the assembled, ranked brief for its owner. | a lead-brief scout (below) |
| Dispatcher / campaign | detection already exists and the job is arming one finding per run for implementation, with a contract a coding agent can execute. | an improve-my-tool campaign scout (below) |
| Fleet meta-scout / reviewer | the watched surface is the scout fleet itself: its configs, runs, reports, and memory. | signals-scout-inbox-validation; a fleet reviewer, a fleet digest (below) |
Anomaly watcher
The default specialist shape, and the one most surfaces fit.
- Watched data: one product surface's metric over time (error counts, log volume, MRR, CSP violations, response rates).
- Discriminator: deviation of the latest complete bucket from a seasonality-matched baseline — and a cheap profile-shape read to triage first (e.g. error tracking's
countvsdistinct_usersratio separates broad-reach bursts from single-user loops). Name the discriminator at the top; it's the whole game. - Dedupe + memory:
dedupe:<domain>:<entity>gates re-filing per entity;pattern:<domain>:baselinerecords what normal looks like so the next run doesn't re-derive it. - Gotcha: score the latest complete bucket, not the in-progress one — a partial current hour/day always looks like a drop.
- Don't reinvent the scoring. When the metric is a saved time-series insight, score it with PostHog's own detectors via
alert-simulaterather than hand-rolling anomaly math — it already handles seasonality and the team's own alert thresholds. Fall back to a hand-computed robust z-score (|value − median| / (1.4826 × MAD)) only when the series isn't a saved insight. - Score the rate, not the raw total. Normalize by the relevant denominator — cost per unit, conversion % per funnel stage, error share — so a legitimate volume change doesn't read as an anomaly (more traffic raises total spend but not cost-per-unit). The "raw total moved" false positive is the most common one here.
- Watch the mix, not only the level — a stable total hides a broken part. Where the metric decomposes into segments (locales, categories, entry methods, products, channels), score each segment's share of the total as its own series alongside the total.
A localized app can lose one language route entirely, a content feed can lose a category to a curation bug, and a physical entry method (a scanned tag, a deep link) can stop working — all while aggregate volume holds, because the remaining segments absorb the traffic and the total-only watcher stays silent through every one of them.
This is the same masked-shift logic
signals-scout-customer-analytics-billing-and-usageapplies per account and product, and it generalizes to any dimension whose members substitute for each other. Two rules stop it firing constantly: require a minimum volume per segment before scoring its share, and score each share against its own trailing baseline rather than an expected even split — segments are legitimately uneven. Apply that floor to the segment's trailing or expected volume, never to the bucket being scored: a segment that has gone to zero fails a current-volume floor and drops out of the sweep, which is exactly the outage the watcher exists to catch. - Contract (SLO) variant. When the team has explicit success-rate contracts — SLOs with error budgets — score against the contract, not a trailing baseline: detect fast burns (an active incident eating the budget now) and slow burns (a rolling success rate creeping below target), SRE-style. Two disciplines change: sweep every watched operation/segment pair systematically each run rather than only the loudest (a quiet pair's budget can be gone before its raw count looks scary), and treat any budget breach as reportable even when the trailing baseline is equally bad — a violated contract is signal by definition. Everything else (dedupe, memory, close-out) is the standard anomaly-watcher shape.
- Split by release surface before calling a trend fleet-wide. A payment decline rate, an exception rate, or a conversion drop that reads as product-wide is often one platform, one app version, or one product line inside a shared stream. Require explicit product, platform, and rollout attribution (iOS vs Android vs web; the current build vs older installs; the new product surface vs the legacy one sharing its events) before filing, and score each surface against its own baseline. The sharpest case is a dual-release architecture: a browser app that updates itself next to a desktop plugin, add-in, or native app people install by hand, so the two run different versions of the same code for weeks. Compare exceptions per session between the two runtime contexts and compare older installed versions against the current one; a divergence there is version skew, not a regression, and the fix is a compatibility shim or an upgrade nudge rather than a rollback.
- Predict first, then measure (active inference) variant. Instead of scoring the latest bucket against a trailing baseline, the scout keeps a world model in its scratchpad: the baselines it believes, a pre-registered prediction for the next window written before the query runs, a ranked list of open hypotheses about what would move the metric, and for each one the observation that would confirm or retire it. The report is the prediction error, not the number: "expected 40–55 healthy signups this week from the baseline plus the pricing-page test; saw 71; the excess is all from one campaign UTM" is a finding, while "71 signups" is a status update. Pair it with a change log of the things that could shift the metric (merged website PRs above a materiality bar, experiments started or stopped, deploys) so a surprise can be attributed rather than re-derived, and post a short plain-language digest on a fixed cadence even when quiet, because a silent predictor gives its reader no way to tell calibration from absence. This is the shape to reach for when the team wants a metric understood rather than merely alarmed on, and when the volume is low enough that per-bucket anomaly scoring is mostly noise.
- Copy the closest specialist verbatim and replace the surface + discriminator.
Read
products/signals/skills/signals-scout-error-tracking/SKILL.mdfor the cleanest worked example (itscount-vs-distinct_userstable is the canonical discriminator).
Liveness / absence watcher
The anomaly watcher's inverse: the signal is an expected event not happening.
This is one of the most common genuinely-new shapes users author for themselves, because almost nothing else in a monitoring stack watches for silence — error tracking only sees code that throws, and the failure here is a 200 OK with the business outcome missing.
- Watched data: an event (or event pair) that should fire — a pipeline stage, a scheduled control, an automation execution, an external callback, your own capture volume.
- Discriminator — absence gated by a heartbeat. Silence alone is ambiguous: "broken" and "nothing to do" look identical.
Pair the watched event with a companion heartbeat that proves the system is otherwise alive, and fire only on heartbeat present, expected event absent.
Naming the heartbeat is the whole design job — without one the scout can't tell an outage from a quiet day.
Override the standard quick close-out.
scout-anatomy.mdtells a scout to writenot-in-use:and stop when the watched event is missing — for this pattern that closes out at the exact moment the finding appears. Gate the early exit on the heartbeat and the recorded cadence, never on the expected event: no heartbeat (or no cadence learned yet) means genuinely not in use; heartbeat present with the expected event missing is the finding. - Two granularities, same discriminator:
- Aggregate silence — one stream goes quiet while its companion keeps firing.
E.g. a scheduled compliance or security check's success event stops appearing while the rest of the pipeline's events continue (the control silently stopped running); an automation/workflow shows
activewith zero executions while its trigger event still has volume (a filter or config change silently dropped 100% of traffic). - Per-item reconciliation ("promise made vs promise kept") — join each antecedent event to its expected consequent within a window, and score the unmatched share against its own baseline. E.g. payment initiated → webhook received; order placed → fulfillment confirmed; an in-product flow started → the third-party fetch that should complete it (a completion-rate cliff with zero exceptions is exactly this shape).
- Aggregate silence — one stream goes quiet while its companion keeps firing.
E.g. a scheduled compliance or security check's success event stops appearing while the rest of the pipeline's events continue (the control silently stopped running); an automation/workflow shows
- Proven variants:
- Compliance / control liveness — the expected event is a security, privacy, or audit control; its absence is a compliance gap by definition, so report even when nothing user-facing broke. The stronger form is a coverage ratio rather than a bare absence: governed operations are supposed to emit an audit record, so score audit events ÷ operational activity (heartbeats, commands executed) and fire when operations continue while the ratio falls toward zero. The system is running untracked, which no error will ever announce. The same ratio logic covers a data-erasure or offboarding workflow (deadline breaches, retry debt, a final-sweep queue that never drains) and identity hygiene (a shared or low-entropy identifier pooling unrelated sources, a machine identity carrying a real person's properties), where the discriminator is identifier entropy against provenance fan-in rather than volume.
- Commitment ledger (promise made by a person, not a system) — the antecedent is something an account or a team said it would do, captured from unstructured sources (a call transcript, a CRM next-steps field, a support thread), mapped to the product signal that would prove it happened, with that signal's baseline snapshotted at capture time.
Each run re-checks the signal and reports only movement: a commitment that landed (with whatever it opens up next) or one that has gone a soak window with a flat signal (with a suggested nudge).
A periodic scoreboard of the open ledger is the digest form.
The memory is the ledger itself,
ledger:<domain>:<account>:<commitment>with the mapped signal, the baseline, and the capture date, and the disqualifier is a commitment with no measurable signal, which stays a note rather than a ledger row. - Automation liveness — the watched entity is a PostHog automation (a workflow, a CDP destination): configured-active with zero successes and zero failures while the trigger has volume is the silently-dark shape a delivery-failure watcher misses.
- Capture / instrumentation liveness (meta-observability) — the watched surface is the project's own event volume: a cliff means the SDK, a consent flow, or a deploy silently stopped collection, and every other scout is now flying blind. Cheap, product-agnostic, and worth considering for any project whose capture is consent-gated. A cliff detector only catches the abrupt case. Under-capture that arrives gradually — adblocker share creeping up, a consent banner change, an SPA route that stopped firing pageviews — never produces a cliff, and the resulting series looks like a real traffic decline to every other scout in the fleet. Catching that needs a second, independent yardstick: a count of the same thing measured somewhere PostHog's SDK isn't in the path (a CDN or edge analytics visitor count, server access logs, an order count from the app database synced into the w