AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

authoring-scouts

Write, edit, and adapt PostHog Signals scouts: scheduled agents that scan a project and report findings. Use to change a canonical scout's scope, thresholds, schedule, or dry-run settings; add a scout for an uncovered surface; record structured output from scheduled LLM scoring; or give feedback through notes. Covers SKILL.md structure, report requirements, follow-up report checks, structured output, duplicate detection, scratchpad memory, scout notes, per-team skills-store and canonical repository edits, and private prompt trials with saved rubrics. Trigger on "write/edit/customize a signals scout", "new scout for X", "tune my scout schedule", "make a scout that watches <event>", "score/judge/measure X with a scout", "test/compare scout prompts", "generate scout rubrics", "structured output from a scout", "scout output to Slack", "leave a note for / give feedback to a scout".

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--authoring-scouts at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/report-checks.md

Report checks

A report says what was true when it was written. A check says what must stay true afterwards, and names a time to test it. The coordinator runs the due checks and appends each verdict to the report as a check_result artefact, so "did the fix hold?" becomes a stored fact instead of something a person has to remember to re-derive.

Checks are written by scout runs and by the report pipeline's research stage. There is no create surface for a person or a task token: the inbox-report-checks-* tools read checks and never write them.

Look for a duplicate first

Always list the report's checks before you write one:

  • Inside a run: posthog:scout-report-check-list {"report_id": "<report>"}. The report_id is the only input.
  • Outside a run: posthog:inbox-report-checks-list (read-only, task:read).

Two reasons this is a hard step, not a courtesy:

  • A report already carrying an open check (pending or active) for the same claim needs no second one. A duplicate measures the same thing twice and puts two verdicts on one report.
  • A report holds at most five open checks. A duplicate spends one of them.

A resolved report often arrives already covered, because the research stage attaches checks of its own while it authors the report.

The two kinds

Monitoring definitions

When the organization's signals-report-monitoring rollout is enabled, config.monitoring can describe a metric target for observations since Monitoring starts. Supply reference_period_minutes for the period the comparison describes and an explicit threshold_scaling: linear for one additive event count or property sum without a formula, or fixed for a target that retains the same bound. Include baseline_period_minutes for the actual period measured whenever you supply baseline_value. Periods are positive whole minutes, at most 366 days. Do not infer scaling from the display format or invent unknown periods. Omit monitoring if these facts are unknown or the rollout is unavailable; the server refuses it while disabled.

For example, a target of at most 70 errors per seven days uses comparison: {"operator": "lte", "value": 70} and monitoring: {"reference_period_minutes": 10080, "threshold_scaling": "linear", "baseline_period_minutes": 20160} when the supplied baseline measures fourteen days. A percentage target uses fixed; its numerical threshold does not shrink with elapsed time. Keep the bounded query and whole-window comparison valid for existing Resolved behavior. This definition adds no Monitoring execution or final decision. The periods do not set the soak, run cadence, expiry, or Monitoring duration.

metric_threshold

One bounded query, one comparison, no sandbox and no LLM. The coordinator measures it in the same tick that collects it, so no scout run is spent.

Config fields:

  • metric_id — a metric already on the report, when one measures the right thing. Its query is copied onto the check when the check is created.
  • query — your own bounded Trends query, when no metric on the report fits. Keep date_from relative (-24h, -13d) and leave date_to empty, so each run measures the window before itself rather than the window before the check was written; absolute dates are refused. The query must produce exactly one output series — one event or action series, or up to ten combined with exactly one formula — with no breakdown and no compare mode.
  • comparison — lte, gte, or between, with the value (or bounds) the measurement must satisfy to pass.
  • baseline_value — what you measured when you wrote the check, stored for the reader.

Send exactly one of metric_id and query.

Use this kind wherever one number settles the claim and an event or action series can carry it, which covers most volume and error-rate reports. Two traps:

  • The comparison is the level a reader accepts as "the problem stopped", not the baseline itself. A lte at the baseline passes on no improvement at all.
  • A lte comparison passes on a measurement of zero, and nothing in this lane notices that the surface simply went quiet. Where zero could equally mean the traffic stopped, make it an agent check and name the denominator to read first.

The check aggregates its whole window into one number at run time, so a window longer than the soak mixes pre-fix traffic into the result and can fail a fix that held. Keep date_from equal to the soak.

agent

For a claim no single number settles, or a number that lives outside events — a log rate, a fix whose effect shows in which entities fire rather than how many, a claim that needs a stack trace read.

Config fields:

  • instructions — what a later run must establish, in your own words. Carry the baseline you measured here: this kind has no baseline_value field.
  • probe_hints — up to five concrete places to look: an issue id, a service name, the query to repeat.
  • skill_name — the scout that answers the check. Leave it unset and the check goes to the fleet's follow-up scout.

The write is refused when the project has no scout that can run the check: no config or no live skill for its lane. Write a metric_threshold check instead. A paused lane still accepts the write.

The check is dispatched as a scout run, which closes it with scout-check-record-result (passed, failed, inconclusive, or errored, plus an explanation). The run records passed or failed when the evidence meets the bar the check states, inconclusive with a reason when its tools worked but the evidence cannot settle the question, and errored only when a tool or model call failed. A run that ends without recording a result is scored errored and retried. So write instructions that state the bar: what a pass looks like, and what a fail looks like.

The text in instructions and probe_hints reaches the run as evidence, not as instructions it obeys.

Scheduling

  • next_run_at — when to look first. Allow deploy and soak time. The API refuses a timestamp that is not in the future, so send a near-future time when the window you reasoned about has already passed. Do not drop the field to avoid the arithmetic: an omitted next_run_at defaults to seven days out, which is usually long past the soak you wanted.
  • run_interval_minutes — leave it unset. One look after the soak is the shape of a check. A level worth re-measuring several times a day is an alert, and the alerts product has the notification and deduplication machinery for it. The floor is six hours, a check runs at most ten times, and its horizon is 90 days.
  • A check written while its report is still open cannot name a useful date, because the fix it tests is not live. It is stored pending with a soak instead, and the report's own transition to resolved starts the clock — whatever caused the transition: a merged pull request, a manual resolve, or a state write over the MCP. The soak defaults to 24 hours.

Statuses and what each verdict does

pending and active are the open states. passed, failed, errored, inconclusive, expired, and cancelled are terminal: a terminal check never reschedules, so a claim still worth watching needs a new check.

  • passed re-arms only a recurring check that has runs left. Otherwise the check is done.
  • failed always retires the check.
  • errored retries. Three consecutive errored runs retire the check as misconfigured.
  • inconclusive does not spend the error budget. Its reason decides what happens next:
    • awaiting_data looks again after 24 hours, then 72 hours, then 7 days. After three looks, or when the next look falls past the horizon, the check ends as inconclusive.
    • unmeasurable, needs_manual_verification, and no_fix_to_measure end the check as inconclusive at once, because waiting does not help.
  • cancelled comes from scout-report-check-cancel, for a check no longer worth running. The coordinator also cancels an agent check on a project with no scout configs, because no run can ever answer it. Verdicts already recorded stay on the report, and a finished check cannot be cancelled.

A failed check on a resolved report always reaches the inbox as a fresh report linked follow_up_of the resolved one. A resolved report is no longer in the inbox, so a verdict left only on its artefact log would reach nobody. Which path files the fresh report depends on the check:

  • metric_threshold: the breach is emitted as a signal, and the pipeline files the report. This lane has no scout in it to author anything.
  • agent, on a lane that holds emit_report: the shared check brief tells the run to author the report itself before it records failed, and to name the new report in the verdict's explanation. This covers every lane, the fallback lane and named lanes alike. The run skips the new report only when the relapse is already in the inbox: an open report it authored about the same fix, which it edits when it holds edit_report, or an open report filed after the fix that already describes the recurrence, which it cites.
  • agent, on a lane without emit_report: the run cannot author, so the verdict is emitted as a signal, the same path a metric breach takes.

A lane's own skill does not need to repeat this rule. It can add to it: the fallback lane, signals-scout-inbox-validation, also sets the report's priority and adds a recurrence_of link when the same problem is still happening. passed, inconclusive, and errored verdicts file nothing new.

Reading a report's checks

inbox-report-checks-list returns every check on a report, newest first, with status, last_outcome, next_run_at, run_interval_minutes, runs_remaining, and expires_at. inbox-report-checks-retrieve returns one check with its full config. query and baseline_value read as null for a caller who cannot read the data they describe.

The verdicts themselves are not on the check row. They are check_result artefacts on the report, so read them through the report's artefact list.

When to attach nothing

A report can be plainly non-measurable: a documentation change, a process recommendation, a one-off data correction. Attach no check, and record why in scratchpad memory instead. Honest unverifiability is worth more than a threshold nobody believes.