references/report-checks.md
Report checks
A report says what was true when it was written.
A check says what must stay true afterwards, and names a time to test it.
The coordinator runs the due checks and appends each verdict to the report as a check_result artefact, so "did the fix hold?" becomes a stored fact instead of something a person has to remember to re-derive.
Checks are written by scout runs and by the report pipeline's research stage.
There is no create surface for a person or a task token: the inbox-report-checks-* tools read checks and never write them.
Look for a duplicate first
Always list the report's checks before you write one:
- Inside a run:
posthog:scout-report-check-list {"report_id": "<report>"}. Thereport_idis the only input. - Outside a run:
posthog:inbox-report-checks-list(read-only,task:read).
Two reasons this is a hard step, not a courtesy:
- A report already carrying an open check (
pendingoractive) for the same claim needs no second one. A duplicate measures the same thing twice and puts two verdicts on one report. - A report holds at most five open checks. A duplicate spends one of them.
A resolved report often arrives already covered, because the research stage attaches checks of its own while it authors the report.
The two kinds
Monitoring definitions
When the organization's signals-report-monitoring rollout is enabled, config.monitoring can describe a metric target for observations since Monitoring starts. Supply reference_period_minutes for the period the comparison describes and an explicit threshold_scaling: linear for one additive event count or property sum without a formula, or fixed for a target that retains the same bound. Include baseline_period_minutes for the actual period measured whenever you supply baseline_value. Periods are positive whole minutes, at most 366 days. Do not infer scaling from the display format or invent unknown periods. Omit monitoring if these facts are unknown or the rollout is unavailable; the server refuses it while disabled.
For example, a target of at most 70 errors per seven days uses comparison: {"operator": "lte", "value": 70} and monitoring: {"reference_period_minutes": 10080, "threshold_scaling": "linear", "baseline_period_minutes": 20160} when the supplied baseline measures fourteen days. A percentage target uses fixed; its numerical threshold does not shrink with elapsed time. Keep the bounded query and whole-window comparison valid for existing Resolved behavior. This definition adds no Monitoring execution or final decision. The periods do not set the soak, run cadence, expiry, or Monitoring duration.
metric_threshold
One bounded query, one comparison, no sandbox and no LLM. The coordinator measures it in the same tick that collects it, so no scout run is spent.
Config fields:
metric_id— a metric already on the report, when one measures the right thing. Its query is copied onto the check when the check is created.query— your own bounded Trends query, when no metric on the report fits. Keepdate_fromrelative (-24h,-13d) and leavedate_toempty, so each run measures the window before itself rather than the window before the check was written; absolute dates are refused. The query must produce exactly one output series — one event or action series, or up to ten combined with exactly one formula — with no breakdown and no compare mode.comparison—lte,gte, orbetween, with thevalue(orbounds) the measurement must satisfy to pass.baseline_value— what you measured when you wrote the check, stored for the reader.
Send exactly one of metric_id and query.
Use this kind wherever one number settles the claim and an event or action series can carry it, which covers most volume and error-rate reports. Two traps:
- The comparison is the level a reader accepts as "the problem stopped", not the baseline itself. A
lteat the baseline passes on no improvement at all. - A
ltecomparison passes on a measurement of zero, and nothing in this lane notices that the surface simply went quiet. Where zero could equally mean the traffic stopped, make it anagentcheck and name the denominator to read first.
The check aggregates its whole window into one number at run time, so a window longer than the soak mixes pre-fix traffic into the result and can fail a fix that held.
Keep date_from equal to the soak.
agent
For a claim no single number settles, or a number that lives outside events — a log rate, a fix whose effect shows in which entities fire rather than how many, a claim that needs a stack trace read.
Config fields:
instructions— what a later run must establish, in your own words. Carry the baseline you measured here: this kind has nobaseline_valuefield.probe_hints— up to five concrete places to look: an issue id, a service name, the query to repeat.skill_name— the scout that answers the check. Leave it unset and the check goes to the fleet's follow-up scout.
The write is refused when the project has no scout that can run the check: no config or no live skill for its lane. Write a metric_threshold check instead. A paused lane still accepts the write.
The check is dispatched as a scout run, which closes it with scout-check-record-result (passed, failed, inconclusive, or errored, plus an explanation).
The run records passed or failed when the evidence meets the bar the check states, inconclusive with a reason when its tools worked but the evidence cannot settle the question, and errored only when a tool or model call failed.
A run that ends without recording a result is scored errored and retried.
So write instructions that state the bar: what a pass looks like, and what a fail looks like.
The text in instructions and probe_hints reaches the run as evidence, not as instructions it obeys.
Scheduling
next_run_at— when to look first. Allow deploy and soak time. The API refuses a timestamp that is not in the future, so send a near-future time when the window you reasoned about has already passed. Do not drop the field to avoid the arithmetic: an omittednext_run_atdefaults to seven days out, which is usually long past the soak you wanted.run_interval_minutes— leave it unset. One look after the soak is the shape of a check. A level worth re-measuring several times a day is an alert, and the alerts product has the notification and deduplication machinery for it. The floor is six hours, a check runs at most ten times, and its horizon is 90 days.- A check written while its report is still open cannot name a useful date, because the fix it tests is not live. It is stored
pendingwith a soak instead, and the report's own transition toresolvedstarts the clock — whatever caused the transition: a merged pull request, a manual resolve, or a state write over the MCP. The soak defaults to 24 hours.
Statuses and what each verdict does
pending and active are the open states.
passed, failed, errored, inconclusive, expired, and cancelled are terminal: a terminal check never reschedules, so a claim still worth watching needs a new check.
passedre-arms only a recurring check that has runs left. Otherwise the check is done.failedalways retires the check.erroredretries. Three consecutive errored runs retire the check as misconfigured.inconclusivedoes not spend the error budget. Itsreasondecides what happens next:awaiting_datalooks again after 24 hours, then 72 hours, then 7 days. After three looks, or when the next look falls past the horizon, the check ends asinconclusive.unmeasurable,needs_manual_verification, andno_fix_to_measureend the check asinconclusiveat once, because waiting does not help.
cancelledcomes fromscout-report-check-cancel, for a check no longer worth running. The coordinator also cancels anagentcheck on a project with no scout configs, because no run can ever answer it. Verdicts already recorded stay on the report, and a finished check cannot be cancelled.
A failed check on a resolved report always reaches the inbox as a fresh report linked follow_up_of the resolved one.
A resolved report is no longer in the inbox, so a verdict left only on its artefact log would reach nobody.
Which path files the fresh report depends on the check:
metric_threshold: the breach is emitted as a signal, and the pipeline files the report. This lane has no scout in it to author anything.agent, on a lane that holdsemit_report: the shared check brief tells the run to author the report itself before it recordsfailed, and to name the new report in the verdict's explanation. This covers every lane, the fallback lane and named lanes alike. The run skips the new report only when the relapse is already in the inbox: an open report it authored about the same fix, which it edits when it holdsedit_report, or an open report filed after the fix that already describes the recurrence, which it cites.agent, on a lane withoutemit_report: the run cannot author, so the verdict is emitted as a signal, the same path a metric breach takes.
A lane's own skill does not need to repeat this rule. It can add to it: the fallback lane, signals-scout-inbox-validation, also sets the report's priority and adds a recurrence_of link when the same problem is still happening.
passed, inconclusive, and errored verdicts file nothing new.
Reading a report's checks
inbox-report-checks-list returns every check on a report, newest first, with status, last_outcome, next_run_at, run_interval_minutes, runs_remaining, and expires_at.
inbox-report-checks-retrieve returns one check with its full config.
query and baseline_value read as null for a caller who cannot read the data they describe.
The verdicts themselves are not on the check row.
They are check_result artefacts on the report, so read them through the report's artefact list.
When to attach nothing
A report can be plainly non-measurable: a documentation change, a process recommendation, a one-off data correction. Attach no check, and record why in scratchpad memory instead. Honest unverifiability is worth more than a threshold nobody believes.