AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

authoring-scouts

Write, edit, and adapt PostHog Signals scouts: scheduled agents that scan a project and report findings. Use to change a canonical scout's scope, thresholds, schedule, or dry-run settings; add a scout for an uncovered surface; record structured output from scheduled LLM scoring; or give feedback through notes. Covers SKILL.md structure, report requirements, follow-up report checks, structured output, duplicate detection, scratchpad memory, scout notes, per-team skills-store and canonical repository edits, and private prompt trials with saved rubrics. Trigger on "write/edit/customize a signals scout", "new scout for X", "tune my scout schedule", "make a scout that watches <event>", "score/judge/measure X with a scout", "test/compare scout prompts", "generate scout rubrics", "structured output from a scout", "scout output to Slack", "leave a note for / give feedback to a scout".

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--authoring-scouts at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/scout-patterns.md · 24000

arehouse). Score the ratio of the two rather than either alone, and treat a persistent drift in that ratio as an instrumentation finding rather than a product one. Three things make this work: hold both sides to the same window and the same definition (a CDN "visit" is not a $pageview), decide up front how you separate a real capture regression from your own comparison job breaking, and expect ratios above 100% on SPAs and other client-side-routing surfaces rather than treating them as failures. Recording the ratio itself each run as a structured-output measurement (below) turns it into a chartable series instead of a judgment repeated from scratch every run. This is the same two-independently-readable-sources logic as the intersection pattern below — here the two sources measure one quantity, and their disagreement is the signal.

  • Release verification / first exposure — an exact-once watcher that a rollout actually reached a real user: watch for the first occurrence of the event+property combination that proves the feature landed. A digest-style exception to "reports are for problems": the scout files at most one report — the landing confirmation, or an overdue alarm once the exposure stays conspicuously absent past a soak window — then retires.
  • Dedupe + memory: absence has no row to key on — dedupe on the stable entity/control id (dedupe:<domain>:<control>, with the ongoing-silence window stored in the value), and keep a report:<domain>:<control> pointer so a persisting absence edits the live report rather than filing a fresh one each run. Record the expected cadence per watched control (pattern:<domain>:cadence:<control>) so the next run knows how long silence must last before it's signal — a single unqualified cadence key gets overwritten by whichever control ran last, and a daily control inherits an hourly threshold.
  • Gotchas:
    • Give the consequent its natural lag. Callbacks, webhooks, and settlement events arrive late; score only windows old enough for the pair to have closed, or every run ends in false alarms.
    • Gate by active hours. Many expected events only fire during business hours or on weekdays — compare silence against the entity's own schedule, not the wall clock.
    • Exact-once shapes must end. A first-exposure watcher that confirmed its event should write an addressed: memory and stop reporting (and its owner should disable it), not re-confirm forever.

Zero-result / unmet-demand watcher

The liveness watcher's close relative, one level down: there the expected event is missing, here the event fires normally and the result inside it is empty. Someone searched and got nothing back, picked a vehicle and no store matched, filtered a marketplace down to no inventory, asked the docs a question that returned no page. Nothing is broken by any conventional reading — the request completed, the funnel step fired, no exception was raised, volume looks normal — so this slips past the anomaly watcher, the funnel scout, and error tracking alike. Teams keep arriving at this shape independently across unrelated verticals, which is usually the sign of a real gap rather than a niche.

  • Watched data: an event representing a request whose payload says how much came back — a result count, a match count, an n_results: 0 flag — together with the properties describing what was asked for (the query terms, the category, the location, the filter combination).
  • Discriminator: the empty-result rate, segmented by the dimension that describes the ask. The aggregate rate is nearly useless — it barely moves, and every product has a steady background of typos and impossible queries. The signal is one slice going empty: this care type in this postcode, this vehicle and tyre size, this category of question. Score each segment against its own trailing baseline, exactly as the anomaly watcher does. Run an absolute lane beside the relative one, or the worst gaps are invisible: a high-demand segment that has always returned nothing has a 100% trailing baseline and never deviates from it, and a newly-introduced segment has no baseline at all. Both are prime supply gaps and both are silent to a purely baseline-relative score, so also flag any segment above an absolute demand-and-emptiness threshold regardless of how it compares to itself.
  • Say which of the two readings you mean, because they go to different people. A zero result is either a supply gap — the catalog, inventory, index, or content genuinely has nothing, and the fix is to go get some — or a matching defect — the supply exists but the query never reached it, through a too-tight filter, a bad geo radius, a stale or half-built index. These are a product decision and a bug respectively, so never file the finding without a call. Cheap corroboration separates them: did this same ask succeed before (a step change points at a defect, a slow climb at demand outgrowing supply), and does a deliberately broadened version of it succeed now (if widening the radius finds plenty, the supply was there)?
  • Rank by demand × emptiness, never emptiness alone. A rare combination at 100% empty matters far less than the most-searched one at 30%, and a scout that sorts on rate alone fills the inbox with the long tail. What a human actually wants out of this pattern is a ranked worklist — the slices where the most people asked and the fewest were served. Count distinct people, not requests. One frustrated person reformulating the same failed search ten times is a single unmet need, and raw request counts rank that retry loop above a gap hitting fifty people. Collapse near-identical retries within a session and score on distinct users or sessions.
  • A volume floor is load-bearing here. Three searches at 100% empty is not a finding; require a minimum number of distinct askers per segment per window before scoring it, and say what the floor is in the body.
  • Dedupe on the segment key, not the query string. dedupe:<domain>:<care-type>:<postcode>, not the raw text someone typed — query strings are unbounded and near-unique, so keying on them refiles forever and never converges. Cap the segments reported per run and roll the remainder into a count. Keep each segment's normal empty-rate in pattern:<domain>:baseline:<segment>, and write addressed: when a gap closes — supply arriving is worth noticing, and worth telling the team their fix landed.
  • Gotcha — the empty case is often not instrumented at all. Plenty of products only capture a result event when there are results, so the zero case is an absence rather than a 0. Confirming the property exists is not enough — read-data-schema happily finds result_count on a stream of successful searches, so the check passes while no zero-valued row can ever reach you. Confirm that result_count = 0 rows actually occur, and sanity-check the event's volume against an independent request or search denominator; a result event that never dips to zero and undercounts the searches you know happened is a one-sided stream, not a healthy one. Either way that is itself the finding: file the instrumentation gap (the recommendation/gap pattern) rather than inferring emptiness from a missing follow-on event, which cannot distinguish "no results" from "user navigated away".
  • Generalizes to any request-with-a-result-set: site and in-app search, a marketplace with no inventory in a location, a filter combination with no matches, an autocomplete with no suggestions, an API lookup returning an empty list. Over a docs or help search it doubles as a content backlog — the questions people ask that you have not answered.

Watchlist explore/exploit

For a surface with more to watch than one run can cover (a busy project's dashboards and insights). The scout can't re-check everything every run, so it curates.

  • Watched data: a durable, scratchpad-held watchlist of high-value entities discovered over time (by view count, dashboard membership, traffic).
  • Discriminator: robust (MAD) deviation from each watched item's own baseline.
  • The balance: each run splits effort between exploit (re-check watchlist items that are due) and explore (discover new high-value items to add). Neither alone is enough — exploit-only goes stale, explore-only never follows up.
  • Dedupe + memory: the watchlist itself is the memory — watchlist:<domain>:<id> entries with last-checked timestamps and per-item baselines. This is the one specialist that bundles its own references; read products/signals/skills/signals-scout-anomaly-detection/ for the full treatment.

Curated (fixed) variant — the common user ask. When the team already knows exactly which entities matter ("watch these dashboards / insights / metrics"), drop the explore half: the watchlist is a fixed, curated set held in the scratchpad (or even inlined in the body), so a run spends almost nothing on discovery and almost everything on "is the latest number worth a human's attention?". This is what most users mean by "keep an eye on my key dashboards", and it's the cleanest first scout to hand someone. Still reconcile the set against reality each run (entities get renamed/deleted), and still score each item against its own seasonality-matched baseline — you've only removed discovery, not scoring. The worked shape: a fixed list of dashboard / insight ids in the scratchpad, scored tile-by-tile via alert-simulate, with the priority items re-checked every run and the rest rotated in as time allows.

Cross-product correlation

The generalist's job. Not a deep dive into one surface — that's what specialists are for — but the seams between surfaces.

  • Watched data: signals from multiple products at once, looking for causal chains: a deploy → an error burst → a conversion dip → a revenue drop.
  • Discriminator: temporal coincidence + a plausible causal story across ≥2 surfaces.
  • Technique: rotate the investigative lens across runs to avoid lens-lock (a generalist that always looks at errors becomes a worse error-tracking specialist). Start from signals-scout-general.

Recommendation / gap

The odd one out: nothing is wrong, but something is missing or sub-optimal. Files P3 recommendations rather than P0–P2 anomalies.

  • Watched data: the delta between what exists and what good practice would have — events with no insight coverage, critical events with no alert, a sequential funnel nobody built, insights pointing at events that stopped firing.
  • Discriminator: a high-value entity that lacks the coverage/configuration it should have.
  • Calibration: default priority P3 with actionability: requires_human_input; weight by how much the gap matters, not by urgency. Don't flood the inbox — a recommendation the team won't act on is noise.
  • See products/signals/skills/signals-scout-observability-gaps/SKILL.md.

Warehouse-backed source scout

The pattern that lets a scout watch anything PostHog can ingest. A non-PostHog source (a Slack channel, a billing system, a CRM, a support tool, a social-listening feed) is synced into the data warehouse on a schedule; the scout reads the resulting table with execute-sql and turns it into signals. The watched surface is not analytics data at all — it's whatever that upstream system produces.

  • Watched data: one (or a few) warehouse tables. Always confirm columns with execute-sql against system.information_schema.columns first — column names are source-defined and often opaque.
  • Discriminator — pre-classified vs derived, and know which you have:
    • Pre-classified — if the upstream tool already labels rows (a sentiment field, a category, a status, a priority), anchor on that. It's a free, high-signal discriminator — e.g. a social-listening feed that ships a per-item sentiment.
    • Derived — most synced sources give you nothing pre-labeled (a raw Slack/Discord channel, a support stream). Build the discriminator from the row's own shape: topic × problem/request language × recurrence, boosted by corroboration (a relayed customer voice, ≥2 people hitting the same thing). This is harder — calibrate it against the inbox more carefully than a pre-classified one.
  • Dedupe + memory: dedupe on a stable source id carried in the row (a post id, a ticket id, an external primary key) — dedupe:<domain>:<source_id>. Don't dedupe on the warehouse row id; syncs re-materialize rows.
  • Gotchas — these bite every warehouse scout:
    • Watermark/cursor. Synced tables are append-only and grow; consecutive syncs often overlap, so the same logical record recurs across rows and across runs. Track how far you've processed in a scratchpad cursor (pattern:<domain>:cursor = "processed through {timestamp}") and only look past it each run. The cheap close-out is "has the max timestamp advanced past my cursor?"
    • Sync lag — anchor on the data, not the wall clock. The sync itself runs behind real time (often hours), so a quiet last hour usually means the sync is lagging, not that the source went silent. Window your queries relative to the table's own max(timestamp), not now(), and don't mistake sync lag for "nothing happening".
    • Timestamp parsing. Warehouse timestamps are often strings — parse explicitly (parseDateTimeBestEffort(...)), and confirm which parse functions the table supports rather than assuming.
    • Threaded / conversational sources — the thread is the unit, not the row. For a Slack or Discord channel, a support thread, or any forum-shaped source, a single row is a tiny fragment ("they", "i made them") meaningless alone. Aggregate to the thread root (e.g. coalesce(thread_ts, ts) for Slack), read the whole thread before judging it, and dedupe on the thread root id, not the message row. A nice touch: reconstruct a permalink back to the source thread from its id so the finding links straight to it.
    • The table may not be in the project profile. It's a warehouse table, not an event, so scout-project-profile-get won't list it. Rely on SQL; handle the "table missing entirely" case with a not-in-use:<domain>:team{team_id} close-out.
    • Evidence citation: cite the source record's id as the evidence source_id so a human can pivot to the original record.
  • Worked example shape — a scout over a Slack channel that's synced to the warehouse: the upstream tool posts pre-classified items into the channel, the channel syncs to a warehouse table every few hours, and the scout (running hourly) sweeps new rows past its cursor, anchors on the pre-classified discriminator, dedupes by the source post id, and files reports for the few that clear the bar. Everything else — the anatomy, the report contract, the four-states classifier — is identical to an events-based scout.

Custom / single-event scout

When one bespoke event captured into PostHog carries the whole signal (a product's own telemetry, a feedback event, a domain-specific action). The event doesn't have to come from a web or mobile app: CI pipelines, server-side jobs, third-party callbacks, and even physical hardware (a device fleet's heartbeat or fault events) all land as ordinary events, and a scout watches them identically.

  • Watched data: one event, confirmed via read-data-schema (the event and the properties you'll filter on — both are team-specific and may be absent).
  • Discriminator: a discriminating property on the event. Pick the one property that separates actionable from noise (a sentiment, a category, a task_completed=false flag) and anchor on it.
  • Corroboration: strengthen a qualitative finding by quantifying blast radius against a second event — e.g. cross-check a complaint about a tool against that tool's error rate over the same window. "Failed on N of M calls" raises confidence far above the raw complaint.
  • Dedupe + memory: dedupe:<domain>:<entity> per recurring issue; pattern:<domain>:baseline for the normal submission rate/mix.
  • Classifier-verdict drift variant. When the bespoke event is a production ML classifier's output (a fraud/spam/moderation verdict with a confidence score), the discriminator is distribution drift: the verdict rate or confidence distribution per segment stepping away from its own baseline — silent model degradation that no exception will ever announce. The strongest corroboration is a second event carrying corrective user feedback: users disagreeing with the verdict at a rising rate turns a distribution shift into a confirmed quality regression. This is distinct from LLM-generation quality (the AI-observability scout's $ai_* territory) — the watched surface is the business verdict, not the model call.

Open-text theme scout

A cross-cutting variation, not a standalone surface: when the watched data is free text (survey open-text responses, feedback submissions, social posts, support messages), the value is in recurring themes, not individual rows.

  • The core rule: aggregate. Emit one themed finding backed by several items, not one finding per item. A stream of one-off complaints erodes the inbox's trust; a single "these 6 submissions all describe X" is actionable.
  • Discriminator: the same root issue appearing across ≥2 items (same category, same complaint shape, same requested feature) — or a single, unusually sharp, concrete item that's worth surfacing at n=1.
  • Dedupe + memory: dedupe:<domain>:<theme-slug> / addressed:<domain>:<theme-slug> gate the theme, not the individual rows. Cite item ids inline so a human can pivot to the source; quote 1–3 representative items only after sanitizing them (see PII gotcha).
  • Gotcha — PII. Free-text sources routinely contain personal or sensitive data (emails, phone numbers, names, account details). Before putting any excerpt in a finding, sanitize it — summarize the claim, redact contact details and identifiers, and prefer the themed paraphrase over a raw quote. Link the source by id rather than copying sensitive text. Never let raw personal data reach a Signals finding. (The signals-scout-surveys scout is the stricter reference here — match its no-PII posture.)
  • This layers onto the warehouse-backed or custom-event patterns — signals-scout-surveys does it over survey open-text; the same shape applies to any text stream.

Adversarial / abuse-concentration scout

Every other pattern watches a system that is indifferent to being watched. This one watches a party who benefits from not being caught — trial-credit farming, scraping, referral and promo fraud, spam signups, multi-accounting to evade a limit — and that changes the design in ways the other patterns never have to think about.

  • Watched data: ordinary product events (signups, trials, redemptions, requests), read through the identifiers several accounts can share rather than through the accounts themselves — a card fingerprint, a device id, an IP or ASN, an email domain or plus-address root, a user agent.
  • Discriminator: concentration on a shared identifier, paired with non-conversion. Legitimate users scatter thinly across those identifiers; an abuser reuses one, because reuse is exactly what makes the abuse cheap to repeat. Concentration alone is not enough — a corporate NAT, a university, or a popular device model all look concentrated — so require the second half: the cluster does the thing that costs you and not the thing that pays you. Many trials on one card and none converting; heavy traffic from one ASN with near-zero engagement depth; many signups from one domain and no activation. Give the cohort time to convert before counting it against them. A cluster signed up this morning has zero conversions because nobody converts that fast, so scoring fresh cohorts turns every launch campaign and every corporate-card rollout into suspected abuse. Score only cohorts past the product's normal conversion lag, and say what window you used.
  • Quantify the leak, because that number is what decides whether anyone acts. "One card, 40 trials, $50 grant each" is actionable in a way "anomalous signup concentration" never is. Put the cost in the summary.
  • Never route an abuse verdict into automated enforcement. The false positive here doesn't cost a wasted review, it revokes a real customer's trial or blocks their access, and they may never tell you. Default to requires_human_input, give the human the cluster and the evidence, and let them act — this is the pattern where the measurement scout's "a grade is now a routing decision" warning applies most sharply.
  • Dedupe on the shared identifier, not the accounts under it. dedupe:<domain>:<card-hash> / :<asn>. Fresh accounts appear under the same root constantly, so keying on accounts refiles the same ring every run and never converges. noise:<domain>:<identifier> is doing heavy lifting on this pattern — corporate NATs, shared office IPs, QA and load-test accounts, legitimate resellers and agencies all concentrate innocently, and an allowlist that accumulates is what keeps the scout usable past its first week. Key on a pseudonym, not the raw identifier. The identifiers this pattern keys on are personal data — IPs, device ids, email roots, card fingerprints — and the scratchpad is durable and readable over MCP, so a raw value written there outlives the finding that needed it. Use a stable keyed hash in every memory key, keep report evidence to sanitized aggregates plus a pivot a human can resolve themselves, and never paste the raw value into a finding.
  • The target adapts, so treat a signature that goes quiet with suspicion. Record the shape you matched in pattern:<domain>:signature. When a previously-firing shape stops, the honest reading is usually that the technique moved rather than that the abuse stopped — say which you believe in the close-out instead of quietly recording success.
  • Seam with the classifier-verdict-drift variant (under the custom / single-event pattern): that one watches your own anti-abuse model's verdicts for silent degradation. This one watches the raw behavior on a surface where no classifier exists yet, and its findings are often the argument for building one.

External-tool / code-review scout

When the judgement comes from running a tool or reading code, not from analytics. The scout reaches out from the sandbox to a public git repo, assesses recently-changed files, and turns the result into P3 recommendations. There are two judge modes:

  • Tool-as-judge — run a deterministic static-analysis CLI and surface what it finds; the tool is the source of truth, the scout just runs it correctly and triages. Confidence is high because the tool is deterministic.
  • Rules-as-judge — fetch a published ruleset/checklist and have the agent read the code and apply the rules with its own judgment. More flexible, lower intrinsic confidence — only report statically-verifiable violations.

Both share the same skeleton:

  • Watched data: files changed in a recent window (e.g. the last 7 days) in a code repo, and the tool/ruleset output over them.
  • Discriminator: a high-impact finding attributed to recent changes — a violation in a file that changed this week. Noise is the pre-existing backlog, low-severity style nits, and anything a sibling scout already reported for the same file.
  • Calibration: P3 recommendations. One finding per file (bundle that file's issues), cap the reports per run (worst offenders first), and cross-check sibling scouts' runs so two code scouts don't double-report the same file.
  • Dedupe + memory: `dedupe: