AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

authoring-scouts

Write, edit, and adapt PostHog Signals scouts: scheduled agents that scan a project and report findings. Use to change a canonical scout's scope, thresholds, schedule, or dry-run settings; add a scout for an uncovered surface; record structured output from scheduled LLM scoring; or give feedback through notes. Covers SKILL.md structure, report requirements, follow-up report checks, structured output, duplicate detection, scratchpad memory, scout notes, per-team skills-store and canonical repository edits, and private prompt trials with saved rubrics. Trigger on "write/edit/customize a signals scout", "new scout for X", "tune my scout schedule", "make a scout that watches <event>", "score/judge/measure X with a scout", "test/compare scout prompts", "generate scout rubrics", "structured output from a scout", "scout output to Slack", "leave a note for / give feedback to a scout".

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--authoring-scouts at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

SKILL.md · 24000

er-analytics-customer-tasksfeature flag. On a project without it the task tools do not exist and the grant does nothing, so do not grant it there. canvas:writereaches every canvas in a public space and every canvas the acting user created: publish, layout, runtime state, and soft-delete of the acting user's own canvases. A scout run cannot create a canvas, because a scout task has no space to file one in. Create the canvas first, name its id in the body, and follow the one-canvas maintainer variant inreferences/scout-patterns.md. Each published version records the run's task, so version history shows which run published it. experiment:writeis the one grant that changes what end users see. It reaches every experiment in the project: create, edit, duplicate, archive, launch, pause, resume, end, and reset. Launch and pause turn the linked feature flag on and off. A scout cannot delete an experiment, copy one to another project, migrate a legacy one, or open a flag cleanup pull request. Withoutfeature_flag:write, which no scout can hold yet, it also cannot ship a variant or change the variants, rollout, or holdout once an experiment has launched or its flag is on. So a scout sets up drafts and ends experiments, and a person ships them. Say in the body which experiments the scout tends, and which lifecycle changes it may make without a person asking. Only the person the scout's runs act as (whoever authored it) or a project admin can set the field, and grants are activity-logged. A scoped API key must itself carry each scope it grants. A granted scout is told in its run prompt which objects it may change, and is asked to name every change in its close-out. The grant is an upper bound: the acting user's own permissions still apply to each object, and the scout reports a refused write rather than retrying it. A dry run (emit: false) never holds the grant, so a scout can be previewed without it changing anything. To audit the writes after a run, see "Auditing what a scout changed" in working-with-scouts`: the changes land in the activity log under the scout's acting user, tagged "via MCP", with no scout or run name on the row. Applies from the scout's next run.

  • output_destinations — defaults to none. When adding Slack to an existing scout, first read output_destinations, then send the full object with every key preserved. Updates replace the object, so sending only slack removes an existing webhook pointer. Set slack to deliver every surfaced report the scout emits (ready or pending_input) to Slack as well as the inbox; a report the judge suppressed stays out of Slack by design: an integration_id for the workspace, plus either a channel (channel_id|#channel-name) or up to five users to DM (member_id|@display-name), never both. A scoped API key or OAuth token needs integration:read and task:read (or the matching write scopes) on top of signal_scout:write to set slack; a minimally scoped credential gets a permission error. thread_reports: true posts a report as a short lead message with the rest split into replies at the summary's section labels, so a long report isn't clipped; it doesn't change how findings post. Slack delivery is a firehose of that one scout's output — no priority filter, no reviewer routing — so it suits a scout whose bar is already tight rather than a chatty one you're still calibrating. A Slack-delivered scout is also exempt from the ignored-reports auto-pause, since consumption there isn't measurable.
  • display_name: the name the UI shows for the scout. Leave blank to use the default derived from the skill name; it never changes skill_name, which stays fixed. The only setting here the nested config on scout-create does not take: set it with a -config-update after creation.
  • model: pins the model the scout's runs use. Leave unset to follow the fleet default, which is what most scouts should do; set it when a scout's job needs a stronger model (long research) or a cheaper one (a frequent, mechanical probe) and say why in the body. Early access: on a project not enrolled in the scouts-model-config preview, any non-null value is rejected with "Choosing a scout model is not available on this project yet.", so only recommend pinning where that flag is on.
  • mcp_gateway_server_ids: MCP store servers (by id) this scout's runs may mount, chosen from the connections members have shared with the whole team. Empty (the default) mounts none of the shared servers. The intent is that only team-shared connections back a scout run, so runs behave the same whoever edits the scout; where MCP gateway enforcement is not yet active on the project, the launch path may still mount the acting user's personal connections, so check the run's mounted servers in its transcript when that matters. Treat it like network_access: it hands the scout third-party tools with whatever access the shared connection carries, changes are activity-logged, and the body should name what the scout uses each server for.
  • repositories — defaults to []: the scout's sandbox holds no checkout, which is right for a scout that only reads the project over MCP. Set ["organization/repository", ...] for a scout that reasons about code, and its sandbox clones each one before the run starts, so the scout can grep the tree, read the layout, and run the project's own build, type check, and tests instead of fetching files one gh api call at a time. Up to 10 per scout, and each must be reachable through the project's GitHub connection — an unreachable name is refused on write rather than surfacing as a clone failure mid-run. The scout's GitHub access stays read-only whether or not it clones, so a listed repository gives it a tree to read and never the ability to push, comment, or open a pull request. Write access for a scout is a separate opt-in that does not exist yet. A multi-repository scout gets each tree on its default branch; there is no per-repository branch selection. Each tree carries the repository's full commit history, so a skill body can run git log, git blame, and --since against it without an unshallow fetch first. Applies from the scout's next run, and changes are activity-logged.
  • tags — free-form labels grouping the fleet, e.g. ["revenue", "on-call"]. Up to 10 per scout, normalized to lowercase kebab-case (On Call → on-call) and deduped. Set them at create time: a scout that lands already grouped saves a follow-up edit, and the desktop app's scout list filters on them. Prefer a tag that already exists on the fleet (-config-list shows every scout's tags) over minting a near-duplicate — revenue and revenue-analytics fragment the same group. A write replaces the set, so send the full desired list, not just the additions. Filter the roster with -config-list's tags parameter (comma-separated, matches a scout carrying any of them).
  • structured_output_schema — defaults to null (channel off). Set a JSON Schema (draft 2020-12, root "type": "object") describing one structured record and the scout gains a third output channel next to reports: each run is shown the schema and told to submit conforming records via scout-record-output (one per run, or one per judged entity — the skill body decides the cardinality and when to record). Records are validated server-side against the schema (all-or-nothing per call) and recorded in the project as $scout_structured_output events with scalar payload keys flattened to output_<key> properties — so a judging/scoring scout's series is chartable in insights directly, and past records are queryable like any events (filter on subject or run_id, break down on output_<key>). The channel also requires emit: a dry-run scout has nowhere to record to, so scout-record-output fails closed for it. Reach for this when the scout's job is a recurring measurement (judge each sampled report good/bad/unsure with a reason, score accounts, classify sessions) rather than surfacing anomalies; keep enums small and add a free-text reason field so the series is breakdown-friendly and auditable. The skill body should say what to sample, how to judge, and what subject to stamp on each record; the schema owns the record shape. Because the records are ordinary events, anything that consumes events can act on one — a workflow or CDP destination triggered on $scout_structured_output, filtered to your skill_name and an output_<key> value, turns a measuring scout into the front half of an automation (route the verdict to a channel, a task, a CRM) with no human in between. The full design treatment — rubric writing, rates-over-scores record shape, rubric versioning, sampling discipline, the seam with reports, what changes once a grade is a routing decision, and the non-judging variants (structured extraction, state snapshots, synthetic telemetry) — is the recurring measurement / LLM-judge pattern in references/scout-patterns.md.

Steering with notes (no authoring needed)

Sometimes you don't want to change the scout — you want to tell it something. That's what scout notes are for: short steering messages any team member (or an agent acting for one) leaves for the fleet, which every run picks up as prior context alongside its scratchpad and run history. Reach for a note instead of an edit when the steer is feedback, a pointer, or context with a shelf life:

  • Feedback on output: "the staging traffic spike you keep flagging is known noise, stop reporting it".
  • A pointer: "dig into the EU signup funnel this week — we think something regressed".
  • Context the scout couldn't know: "we shipped a new checkout on Tuesday, treat conversion shifts after that as expected".

The tools (reads on the public signal_scout:read scope; because scouts read notes verbatim, writing or deleting one requires the same authorization as editing a scout's skill — the llm_skill:write scope plus skill editor access):

  • posthog:scout-notes-create {"content": "...", "skill_name": "signals-scout-web-analytics"} — address one scout by its exact skill name (roster via scout-config-list; the skill must already exist, so a typo'd target is rejected instead of silently steering no one), or omit skill_name for a general note every scout sees. Optionally set expires_at so a time-boxed note ("watch closely this week") retires itself. skill_name: "pipeline:report-research" addresses the report pipeline's research stage instead of any scout, for guidance about how the reports it builds from clustered signals get researched, judged, and routed ("route billing-adjacent reports to the billing folks"). Reports a scout authors directly never pass through that stage, so a rule for them belongs in a fleet-wide note (omit skill_name) or a per-scout one. It is the only pipeline:* audience; any other value is rejected.
  • posthog:scout-notes-list — browse the active notes; pass skill_name to see what a given scout will read. Expect system-derived notes alongside the human ones: a dismissal or snooze note, a Discuss question, a thumbs-rating note, and a reviewer add/remove on a report are each forwarded to the relevant scout as a note, labelled by origin (report_dismissal, report_discussion, report_feedback, report_reviewer_correction; a note left directly is human). Derived notes expire on their own after ~30 days. Three caveats: a Discuss question or rating note only forwards when the person had the notes-write authorization themselves (skill-editor access plus, on a scoped credential, signal_scout:write and llm_skill:write); a reviewer correction only forwards when the person had skill-editor access on the canonical project (the reviewer change itself still lands on the report); and the list hides every derived origin from a caller without report read access (task:read), since they quote report content. A missing derived note can mean either. A dismissal note is still on the report and a Discuss question on its discussion task, but a rating note exists only as the forwarded scout note (plus an analytics event), so a rating note that did not forward is gone.
  • posthog:scout-notes-delete {"id": "..."} — retire a note that's been acted on or no longer applies.
  • posthog:scout-run-now {"id": <config_id>, "note": "..."}: steer one run only. The note (up to 1,000 characters) is read by that run next to the durable notes and is never delivered to a later run as a note, so use it for "check this now" instead of a note every scheduled run would keep reading. It does stay visible in that run's metadata (run_note) when a later run reads its history, so phrase it as a dated one-off ("today only: ...") rather than standing policy. It needs skill-editor access and, on a scoped credential, llm_skill:write on top of the base signal_scout:write, the same two scopes as a durable note (a 403 otherwise; drop the note to run without it), and spends a run like any manual dispatch.

How scouts treat notes: every run reads its notes in step 1 and is told to let a fresh note visibly shape what it investigates — but notes are advisory. They direct attention; they don't lower the scout's evidence bar or force a report, so a note saying "report X" still gets an honest investigation, not an automatic emit. The scout closes the loop in its run summary (which notes it acted on and how) and folds absorbed guidance into its scratchpad.

Choosing between a note and an edit: a note is the right tool for this project, right now steering and for trying a nudge before committing to it; a skill edit is the right tool once the steer is permanent policy (a disqualifier, a threshold, a scope change). A note that you keep re-leaving is a skill edit waiting to happen — promote it. Note lifecycle stays with humans: scouts never delete notes, so retire acted-on notes yourself (or set expires_at up front) to keep the channel high-signal.

Workflow-triggered runs

A workflow's "Run scout" step starts a run when an event fires, for example a new GitHub issue or a pull request. Every run's prompt names how it started in Your run identity: triggered_by is schedule, manual, workflow, or check (experiment on a private trial run). A workflow run also carries trigger_workflow_id, the id of the workflow that fired. Branch on that value in the body ("if triggered_by is workflow ...") rather than on whether a note is present.

What the run sees depends on the step's note:

  • With a note, the harness makes the subject the note names the run's first job. The run handles that subject completely before anything else: it investigates, decides, files a report or records why it rejected the subject, and writes the memory the body asks for. The scout's routine is optional on that run. If the body says nothing about workflow-triggered runs, the run skips the routine and leaves it to the schedule.
  • Without a note, the run is told a workflow started it and that nothing names the event. It runs the body as it would on a schedule, unless the body says what to do on a workflow-triggered run.

Write a workflow-triggered section in the body when the default is wrong for the scout. For example, "on a workflow-triggered run, also refresh the label cache", or "on a workflow-triggered run with no note, only check the last hour".

Incremental state needs a full pass. A run that skipped its routine or cut it short must leave cursors, baselines, and "last swept" markers as they were, and write memory only about the subject it handled. The harness tells every workflow run with a note this. Name the scout's own incremental keys in the body, because only the body knows which scratchpad entries a partial pass must not move.

Write the step's note from identifiers. Use a pull request number, an issue URL, or an alert id, for example Issue #{event.properties.issue_number}. Do not paste a title, a body, or other free text from the event. The scout reads the note as untrusted input, and an identifier gives it something to look up with its own tools.

Test loop

Dogfood the scout yourself before you ever spend a real run. You — the agent authoring the scout — have the same PostHog MCP tools a scout uses at runtime (execute-sql, read-data-schema, the per-product list tools, scout-project-profile-get). The cheapest, fastest iteration doesn't touch a scout run at all: walk the scout's own logic against the live project by hand. Confirm the watched event/entity exists and has the shape you assumed, run the discriminator to check it actually separates signal from noise on this project's data, and run each explore pattern's queries to see what they surface. This loop is free and instant — refine the body against what you find, re-run the queries, repeat, until the scout's logic holds up on real data. This is where the real iteration happens.

A few runtime tools cannot be dogfooded, so validate them in the first real run instead. Anything that takes a run_id only works from inside a run, and a tool the project has not been switched on for fails whoever calls it. posthog:scout-lighthouse-audit is the one to plan around: it loads a page in a real throttled browser and returns the lab metrics, the element the browser chose as the Largest Contentful Paint, where the LCP time went phase by phase, and the ranked savings estimates — which is how a page-performance finding names a cause rather than guessing one. It needs the run's run_id, it is restricted to an allowlist of public pages (the browser signs in to nothing, so a page behind a login would report the login screen's numbers as the page's), it is limited to the projects switched on for it, and it is capped at five audits per run. A rejected call costs nothing, but once the page loads the slot is spent whatever the result, and every error message ends with how many audits the run has left, which tells a rejection apart from an exhausted budget. So: treat it as available only once a run's own call succeeds, never write a body whose finding depends on it, and keep lab and field evidence labelled apart — one throttled cold load explains a finding, and the field percentile over real users is what establishes there is one. Then check the first real run's transcript for the call and what it returned, the same way you would check a query you could not run by hand.

Compare variants privately

The rubric and private-trial MCP tools currently require staff access and the scout-trials feature flag. Use this path when the user wants to test an existing scout, try an idea, or compare several prompts, models, or reasoning efforts. The trial runs real scouts against project data, captures their reports and memory privately, judges them against one saved rubric, and saves a comparison report. Generation, scout runs, and judging spend model usage; choose the number of variants and repeats within the requested budget.

  1. Choose the source. For an existing scout, get its config from posthog:scout-config-list and read its instructions with posthog:skill-get. For a new idea, create the complete scout with posthog:scout-create, including config: {"enabled": false} and any bundled files. Then call posthog:scout-trial-setup with the config id to check readiness and discover supported models and efforts.
  2. Review and save the rubric. Read posthog:scout-rubric-get. If it needs generation, use posthog:scout-rubric-generate, then poll posthog:scout-rubric-get until generation finishes. Review the suggestions and captured source, select criteria that measure the intended job, and use posthog:scout-rubric-save with the current revision, complete criteria list, and adopt_generation_id when adopting that generation's source. If suggestions are poor, edit the criteria directly: enable or disable any check, revise its wording, add custom checks, or remove unwanted custom/generated checks from the saved list. Keep the six built-in defaults, disabling them when unwanted; omit adopt_generation_id to preserve the saved reference. Generation alone does not save or adopt anything; the saved rubric needs an adopted source before a trial can start.
  3. Start the comparison. Use posthog:scout-trial-start with a baseline and the requested variants, keeping model and effort fixed when comparing prompt changes. Each candidate supplies its full skill_body; the saved source remains unchanged. A request for 5–10 prompts fits one comparison; use distinct hypotheses and avoid multiplying repeats without a reason.
  4. Read the result. Poll posthog:scout-trial-report for the saved comparison, including criterion evidence, failures, applicability, and cost. Resume an interrupted trial with posthog:scout-trial-resume when its error permits it; use posthog:scout-trial-list to find saved trials. Read the evidence behind a score before recommending or applying a candidate.

Read references/lifecycle-and-testing.md for the exact save/adoption semantics, request identities, variant inputs, and result interpretation. If setup is blocked or the tools are unavailable, report the limitation; an ordinary run has different effects and is not a substitute for a private comparison. The existing posthog:scout-trial-create / posthog:scout-trial-get pair runs and reads a single private scout without automatic judging; use scout-trial-start / scout-trial-report for this complete test loop.

Apply changes and run normally

When applying a candidate is part of the user's request, update the saved skill through posthog:skill-update with its current base_version. Use posthog:scout-config-update to set the intended model, schedule, or enabled=true as appropriate to the requested rollout; the agent can enable or pause a scout through this existing tool. This tool cannot save reasoning_effort: a scout with a configured model uses its runtime's default effort, which may differ from the trial. Before enabling a candidate as the tested winner, verify that its ordinary effort matches the trial. If it differs or is unknown, report the effort as unapplied and validate the configuration that ordinary runs will actually use first. A request only to compare candidates leaves the source definition and schedule unchanged.

For an ordinary execution, posthog:scout-run-now {"id": <config_id>} runs the saved scout immediately, even while paused, without changing its schedule or last_run_at. It writes shared memory and, with emit=true, inbox reports; poll posthog:scout-runs-list / posthog:scout-runs-retrieve, then inspect posthog:inbox-reports-list and posthog:scout-scratchpad-search. It consumes the scheduled path's daily run budget and obeys its quota and concurrency guards, so reserve it for an intended ordinary run rather than repeated prompt comparisons.

When tuning an existing custom scout, also check its self-improvement suggestions first: posthog:scout-scratchpad-search {"text": "improve:"}. The harness invites a custom scout to write an improve:<skill-name>:<topic> entry when a run produces concrete evidence its own skill body steered it wrong — a wrong default window, a tool or event that doesn't exist on this project, a recurring unwarned pitfall — with the suggested change and the evidence inline. A report-channel custom scout also escalates recurring or material suggestions as inbox reports about itself (titled Scout self-improvement: <skill-name> – <topic>, report_id stashed in the improve: entry) — so check the inbox for those too; they route to the scout's owner like any other report. An entry re-confirmed across sev