SKILL.md · 24000
ing diff is intended.
- Omit
approve_all(default false) to finalize a run you've already reviewed snapshot-by-snapshot. Finalize is all-or-nothing: it fails with409 not_fully_resolved(and lists what's left) unless every changed/new snapshot is approved, tolerated, or quarantined. - It commits exactly the snapshots approved in the DB — tolerated snapshots keep their baseline and are never overwritten. When a baseline commit is pushed, its SHA comes back on the run's
metadata.baseline_commit_sha. It's absent when nothing needed committing (everything resolved by toleration/quarantine — finalize posts a green status, and the required check turns green when its job re-runs) or when the commit was skipped: no PR, or a409 sha_mismatchbecause the PR has newer commits (that one leaves the gate red — re-run CI on the latest commit and finalize again).
If finalize fails with 409 stale_run, the run has been superseded — visual-review-runs-list { pr_number } and finalize the newest one. A successful finalize often kicks off a fresh CI run, which is normal.
The finalize gate
Finalize is the one irreversible, outward-facing action in this skill: it rewrites the baseline committed to the PR and greens the merge gate. Treat it like pushing to someone's branch — never automatic.
Before any finalize-create call, all of these must hold:
- You verified the diffs. You pulled the current (and, for
changed, baseline) PNGs and looked at them, ran the flake check on anything suspect, and reached a per-snapshot verdict. Metadata alone is never enough. - You presented the verdict and waited. Show the user, per snapshot, what changed and your recommendation, then stop.
- The user explicitly approved this run. A broad "get the gate green" / "fix the PR" is permission to investigate and recommend — NOT to finalize. When the task implies finalizing but the human hasn't said it for this specific run, ask.
approve-create, tolerate-create, quarantine-create and recompute-create don't need this gate when Decide first allows them.
The moment you're about to finalize-create and can't point to a specific human "yes" for this run, stop and ask.
Vocabulary cheat sheet
These appear in tool output and matter for interpretation:
- Run
review_state:needs_review(open, awaiting human),clean(zero diffs),processing(CI still uploading),stale(a newer run on the same PR has superseded this one — checksuperseded_by_id). - Run
run_type:storybook(component snapshots) orplaywright(full-page e2e snapshots). - Snapshot
result:unchanged,changed(real diff),new(no baseline yet),removed. - Snapshot
classification_reason:tolerated_hash(matches a known-tolerated hash, no action needed),below_threshold(under the noise floor),exact(byte-identical),""(real diff requiring review). - Snapshot
review_state:pendingorapproved. - Run
summary:total / changed / new / removed / unchanged / unresolved / tolerated_matched—unresolvedis what's actually blocking review.
Workflows
"What's the VR status of this PR?"
The single most common job. Map a PR number to its run state in two calls.
First confirm the PR is not from a fork.
A fork PR has no run, and visual-review-runs-list returns an empty list for it.
posthog:visual-review-runs-list { pr_number: <n>, limit: 5 }— sort bycreated_atdesc, take the latest non-stale one.- If the run has
summary.changed > 0orsummary.unresolved > 0, drill in:posthog:visual-review-runs-snapshots-list { id: <run_id>, exclude_unchanged: true }and report thechangedsnapshots.
Report back: PR number, run UUID, review_state, summary counts, and the _posthogUrl deep link so the
user can click straight to the diff viewer.
"Is the diff real or unrelated?"
The most useful judgment a code-aware agent can add. Combine three signals: scope match, flake history, and the actual rendered images. The agent should look at the screenshots — not just describe metadata.
-
Scope check —
git diff <base>...HEAD --statagainst the PR's base branch → list of touched paths. Cross-reference withposthog:visual-review-runs-snapshots-list { id, exclude_unchanged: true }filtered toresult: changed→ story identifiers. A Storybook identifier is the kebab-case story title,--, the kebab-case export name, and the theme, for examplescenes-app-settings-user--settings-user-profile--dark. Search the repo's story files for the title (Scenes-App/Settings/User) to find the source path. -
Visual inspection — for each
changedsnapshot, the tool result containscurrent_artifact.download_urlandbaseline_artifact.download_url. These are pre-signed S3 URLs to PNG files; pull them and look:curl -s -o /tmp/vr-baseline.png "<baseline_artifact.download_url>" curl -s -o /tmp/vr-current.png "<current_artifact.download_url>"Then
Readboth files (the Read tool renders images visually) and compare. Things to call out:- The actual visible delta (text changed, button moved, layout shift, color drift, missing element).
- Whether the change is consistent with the diff_pixel_count and diff_percentage in the metadata (e.g. 54% diff but the images look near-identical → screenshot framing changed, not the UI).
- Whether the baseline and current have different dimensions (
width/heightfields). Mismatched dimensions usually mean the story rendered to a different viewport or didn't fully render before screenshot — a flake signal, not a regression, and never render noise to tolerate. change_kind:pixelfor scattered pixel noise,structuralfor a perceptual change. A small diff percentage is often a real structural change.
-
Flake history — run the flake check below for any story that looks suspect.
-
Verdict — combine all three:
- Scope plausible + visible regression matches the code change → real diff, recommend approval.
- Scope mismatch + the story flakes on the default branch → quarantine it, per Decide first.
- Scope plausible + visible regression looks unintended → push a fix; do not approve.
Always include a one-line description of what you saw in the images — the user uses this to decide whether to trust your verdict without opening the VR UI themselves.
Flake check: "Has this story been changing?"
Once you have a suspect snapshot row from visual-review-runs-snapshots-list, ask two separate questions.
Is the story unstable? Only the flakiness overview answers this. Read the repo id with
posthog:visual-review-runs-retrieve { id: <run_id> }. Then call
posthog:visual-review-repos-flakiness-retrieve { id: <repo_id> }, and find the entry whose identifier and
run_type match your snapshot. That entry carries the flake signal:
flakiness_state:broken,unstable,at_risk, orclean.hard_rateandhard_count: the share and number of recent default-branch runs that failed the gate.last_flaked_at: the latest of them.soft_rate: the share that a toleration absorbed.window_runs: the repo's default-branch runs behind those rates. It is the same for every entry of a run type and says nothing about the story.
A story with no entry is quiet only when the response has truncated: false.
With truncated: true, an absent story can be in any state, so absence proves nothing.
Do not tolerate or quarantine on absence alone: check visual-review-runs-tolerated-hashes-list and the snapshot history, or report it.
Did the baseline move? Call
posthog:visual-review-runs-snapshot-history-list { id: <run_id>, identifier: <identifier> }. It returns one row for
each baseline transition on the default branch, not a run-by-run outcome list. It drops a feature branch, and it
collapses consecutive runs that share a baseline. You therefore cannot count outcomes with it.
Both parameters are required. Copy them from the snapshot row: its run_id goes in id, and its identifier goes in
identifier. The snapshot's own id is not a run id, and a call that sends it fails.
visual-review-runs-tolerated-hashes-list takes the same two parameters.
Verdicts:
flakiness_stateisclean, or the story has no entry in an untruncated list → likely a real regression caused by this PR.flakiness_stateisunstablewithhard_count≥ 5 and a recentlast_flaked_at→ flaky story; quarantine it. One failure is enough forunstable, and a single failure is often a real change that merged, so trust the count.flakiness_stateisbroken→ the baseline is wrong, not the story. Do not quarantine; recommend a re-baseline.at_risknever fails a run, so it needs no action on a PR.- Recent
removed, a large-jump dimension change, or a baseline that last moved long ago → baseline likely stale; recommend re-baselining on master.
Triaging the queue
When the user is doing housekeeping rather than asking about a specific PR:
posthog:visual-review-runs-counts-retrieve→ total queue size.posthog:visual-review-runs-list { review_state: needs_review, limit: 50 }(paginate if needed).- Group by
branchauthor orrun_typeto surface clusters (e.g., "12 PRs blocked on the same shared component change" usually means a single underlying root cause to address). - Prefer surfacing runs whose
summary.changed > 0over runs that are onlynew—newmeans no baseline yet, which is usually trivial to approve;changedis the real review work. - Lift stale quarantines: entries in
visual-review-repos-flakiness-retrievewithneeds_decision: truestopped failing or expire soon. Lift one withposthog:visual-review-repos-quarantine-expire-create { id, run_type, identifier }only when it had no hard failure in the window, or a merged fix removed the cause. Before a lift, check that the default branch renders the story as its entry now. Use the latest completed default-branch run of the quarantine's run type whose commit contains the fix, and list its changed snapshots withinclude_quarantined: true, exclude_unchanged: true, followingnext. Lift only when the story is not in that list on several such runs. A rare flake renders clean most of the time, so one clean run is not enough, and a pending run or one of another run type proves nothing. Thebrokenstate alone does not decide it. The state covers 7 days, so it staysbrokenfor days after a fix lands. When the story is still in that list, check that recent default-branch runs render the same changed picture before you re-baseline it. One changed render of a flaky story is a flake, and re-baselining it only swaps which variant fails. Keep the quarantine then. For the same picture on run after run, re-baseline it first, then lift: open a PR that renders the story, approve the snapshot on that run by identifier, finalize the run, and merge the PR. Report a quarantine in that condition before its expiry date, because the expiry fails runs the same way.
Output expectations
For PR-status questions, lead with the verdict in one line, then 2-4 bullets of supporting context. Always
include the _posthogUrl deep link to the run — humans need to see the rendered images to make the call,
the agent can only describe the metadata.
For triage / aggregate questions, a short table beats prose. Group by what the user is going to act on.
What NOT to do
- Do not tolerate to get past a gate, and do not quarantine a diff your own change caused or a
brokenentry. - Do not read a green gate as proof that your change left quarantined stories alone. See Quarantined stories.
- Do not assume the failing GitHub check on a PR is unrelated to VR — if a
visual-reviewcheck is red on a PR you're working on, that's the trigger to run this skill. - Do not read an empty run list on a fork PR as a broken or pending run.
Check
isCrossRepositoryfirst, then triage the offline check instead. - Do not declare a verdict from metadata alone when
result: changed. Pull the baseline and current PNGs and look at them; metadata can only say "something changed", not whether the change is intended.