AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

triaging-visual-review-runs

Inspects PostHog Visual Review (VR) runs that gate PR merges with screenshot regression checks. Use when the user mentions "visual review", "VR", "snapshot diff", "screenshot test", "storybook regression", "playwright snapshot", "quarantine", asks why a PR is blocked or what changed visually, wants to triage the VR backlog, decide whether a snapshot diff is real vs flaky, quarantine or lift a flaky story, or check whether a story has been changing across runs. Also invoke when a PR has a failing `visual-review` status check, when a PR comment mentions "Visual review", when a PR from a fork fails a screenshot check, or when the user is on a branch with an open VR run.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--triaging-visual-review-runs at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 1

SKILL.md · 24000

ing diff is intended.

  • Omit approve_all (default false) to finalize a run you've already reviewed snapshot-by-snapshot. Finalize is all-or-nothing: it fails with 409 not_fully_resolved (and lists what's left) unless every changed/new snapshot is approved, tolerated, or quarantined.
  • It commits exactly the snapshots approved in the DB — tolerated snapshots keep their baseline and are never overwritten. When a baseline commit is pushed, its SHA comes back on the run's metadata.baseline_commit_sha. It's absent when nothing needed committing (everything resolved by toleration/quarantine — finalize posts a green status, and the required check turns green when its job re-runs) or when the commit was skipped: no PR, or a 409 sha_mismatch because the PR has newer commits (that one leaves the gate red — re-run CI on the latest commit and finalize again).

If finalize fails with 409 stale_run, the run has been superseded — visual-review-runs-list { pr_number } and finalize the newest one. A successful finalize often kicks off a fresh CI run, which is normal.

The finalize gate

Finalize is the one irreversible, outward-facing action in this skill: it rewrites the baseline committed to the PR and greens the merge gate. Treat it like pushing to someone's branch — never automatic.

Before any finalize-create call, all of these must hold:

  1. You verified the diffs. You pulled the current (and, for changed, baseline) PNGs and looked at them, ran the flake check on anything suspect, and reached a per-snapshot verdict. Metadata alone is never enough.
  2. You presented the verdict and waited. Show the user, per snapshot, what changed and your recommendation, then stop.
  3. The user explicitly approved this run. A broad "get the gate green" / "fix the PR" is permission to investigate and recommend — NOT to finalize. When the task implies finalizing but the human hasn't said it for this specific run, ask.

approve-create, tolerate-create, quarantine-create and recompute-create don't need this gate when Decide first allows them. The moment you're about to finalize-create and can't point to a specific human "yes" for this run, stop and ask.

Vocabulary cheat sheet

These appear in tool output and matter for interpretation:

  • Run review_state: needs_review (open, awaiting human), clean (zero diffs), processing (CI still uploading), stale (a newer run on the same PR has superseded this one — check superseded_by_id).
  • Run run_type: storybook (component snapshots) or playwright (full-page e2e snapshots).
  • Snapshot result: unchanged, changed (real diff), new (no baseline yet), removed.
  • Snapshot classification_reason: tolerated_hash (matches a known-tolerated hash, no action needed), below_threshold (under the noise floor), exact (byte-identical), "" (real diff requiring review).
  • Snapshot review_state: pending or approved.
  • Run summary: total / changed / new / removed / unchanged / unresolved / tolerated_matched — unresolved is what's actually blocking review.

Workflows

"What's the VR status of this PR?"

The single most common job. Map a PR number to its run state in two calls. First confirm the PR is not from a fork. A fork PR has no run, and visual-review-runs-list returns an empty list for it.

  1. posthog:visual-review-runs-list { pr_number: <n>, limit: 5 } — sort by created_at desc, take the latest non-stale one.
  2. If the run has summary.changed > 0 or summary.unresolved > 0, drill in: posthog:visual-review-runs-snapshots-list { id: <run_id>, exclude_unchanged: true } and report the changed snapshots.

Report back: PR number, run UUID, review_state, summary counts, and the _posthogUrl deep link so the user can click straight to the diff viewer.

"Is the diff real or unrelated?"

The most useful judgment a code-aware agent can add. Combine three signals: scope match, flake history, and the actual rendered images. The agent should look at the screenshots — not just describe metadata.

  1. Scope check — git diff <base>...HEAD --stat against the PR's base branch → list of touched paths. Cross-reference with posthog:visual-review-runs-snapshots-list { id, exclude_unchanged: true } filtered to result: changed → story identifiers. A Storybook identifier is the kebab-case story title, --, the kebab-case export name, and the theme, for example scenes-app-settings-user--settings-user-profile--dark. Search the repo's story files for the title (Scenes-App/Settings/User) to find the source path.

  2. Visual inspection — for each changed snapshot, the tool result contains current_artifact.download_url and baseline_artifact.download_url. These are pre-signed S3 URLs to PNG files; pull them and look:

    curl -s -o /tmp/vr-baseline.png "<baseline_artifact.download_url>"
    curl -s -o /tmp/vr-current.png "<current_artifact.download_url>"
    

    Then Read both files (the Read tool renders images visually) and compare. Things to call out:

    • The actual visible delta (text changed, button moved, layout shift, color drift, missing element).
    • Whether the change is consistent with the diff_pixel_count and diff_percentage in the metadata (e.g. 54% diff but the images look near-identical → screenshot framing changed, not the UI).
    • Whether the baseline and current have different dimensions (width / height fields). Mismatched dimensions usually mean the story rendered to a different viewport or didn't fully render before screenshot — a flake signal, not a regression, and never render noise to tolerate.
    • change_kind: pixel for scattered pixel noise, structural for a perceptual change. A small diff percentage is often a real structural change.
  3. Flake history — run the flake check below for any story that looks suspect.

  4. Verdict — combine all three:

    • Scope plausible + visible regression matches the code change → real diff, recommend approval.
    • Scope mismatch + the story flakes on the default branch → quarantine it, per Decide first.
    • Scope plausible + visible regression looks unintended → push a fix; do not approve.

Always include a one-line description of what you saw in the images — the user uses this to decide whether to trust your verdict without opening the VR UI themselves.

Flake check: "Has this story been changing?"

Once you have a suspect snapshot row from visual-review-runs-snapshots-list, ask two separate questions.

Is the story unstable? Only the flakiness overview answers this. Read the repo id with posthog:visual-review-runs-retrieve { id: <run_id> }. Then call posthog:visual-review-repos-flakiness-retrieve { id: <repo_id> }, and find the entry whose identifier and run_type match your snapshot. That entry carries the flake signal:

  • flakiness_state: broken, unstable, at_risk, or clean.
  • hard_rate and hard_count: the share and number of recent default-branch runs that failed the gate. last_flaked_at: the latest of them.
  • soft_rate: the share that a toleration absorbed.
  • window_runs: the repo's default-branch runs behind those rates. It is the same for every entry of a run type and says nothing about the story.

A story with no entry is quiet only when the response has truncated: false. With truncated: true, an absent story can be in any state, so absence proves nothing. Do not tolerate or quarantine on absence alone: check visual-review-runs-tolerated-hashes-list and the snapshot history, or report it.

Did the baseline move? Call posthog:visual-review-runs-snapshot-history-list { id: <run_id>, identifier: <identifier> }. It returns one row for each baseline transition on the default branch, not a run-by-run outcome list. It drops a feature branch, and it collapses consecutive runs that share a baseline. You therefore cannot count outcomes with it.

Both parameters are required. Copy them from the snapshot row: its run_id goes in id, and its identifier goes in identifier. The snapshot's own id is not a run id, and a call that sends it fails. visual-review-runs-tolerated-hashes-list takes the same two parameters.

Verdicts:

  • flakiness_state is clean, or the story has no entry in an untruncated list → likely a real regression caused by this PR.
  • flakiness_state is unstable with hard_count ≥ 5 and a recent last_flaked_at → flaky story; quarantine it. One failure is enough for unstable, and a single failure is often a real change that merged, so trust the count.
  • flakiness_state is broken → the baseline is wrong, not the story. Do not quarantine; recommend a re-baseline.
  • at_risk never fails a run, so it needs no action on a PR.
  • Recent removed, a large-jump dimension change, or a baseline that last moved long ago → baseline likely stale; recommend re-baselining on master.

Triaging the queue

When the user is doing housekeeping rather than asking about a specific PR:

  1. posthog:visual-review-runs-counts-retrieve → total queue size.
  2. posthog:visual-review-runs-list { review_state: needs_review, limit: 50 } (paginate if needed).
  3. Group by branch author or run_type to surface clusters (e.g., "12 PRs blocked on the same shared component change" usually means a single underlying root cause to address).
  4. Prefer surfacing runs whose summary.changed > 0 over runs that are only new — new means no baseline yet, which is usually trivial to approve; changed is the real review work.
  5. Lift stale quarantines: entries in visual-review-repos-flakiness-retrieve with needs_decision: true stopped failing or expire soon. Lift one with posthog:visual-review-repos-quarantine-expire-create { id, run_type, identifier } only when it had no hard failure in the window, or a merged fix removed the cause. Before a lift, check that the default branch renders the story as its entry now. Use the latest completed default-branch run of the quarantine's run type whose commit contains the fix, and list its changed snapshots with include_quarantined: true, exclude_unchanged: true, following next. Lift only when the story is not in that list on several such runs. A rare flake renders clean most of the time, so one clean run is not enough, and a pending run or one of another run type proves nothing. The broken state alone does not decide it. The state covers 7 days, so it stays broken for days after a fix lands. When the story is still in that list, check that recent default-branch runs render the same changed picture before you re-baseline it. One changed render of a flaky story is a flake, and re-baselining it only swaps which variant fails. Keep the quarantine then. For the same picture on run after run, re-baseline it first, then lift: open a PR that renders the story, approve the snapshot on that run by identifier, finalize the run, and merge the PR. Report a quarantine in that condition before its expiry date, because the expiry fails runs the same way.

Output expectations

For PR-status questions, lead with the verdict in one line, then 2-4 bullets of supporting context. Always include the _posthogUrl deep link to the run — humans need to see the rendered images to make the call, the agent can only describe the metadata.

For triage / aggregate questions, a short table beats prose. Group by what the user is going to act on.

What NOT to do

  • Do not tolerate to get past a gate, and do not quarantine a diff your own change caused or a broken entry.
  • Do not read a green gate as proof that your change left quarantined stories alone. See Quarantined stories.
  • Do not assume the failing GitHub check on a PR is unrelated to VR — if a visual-review check is red on a PR you're working on, that's the trigger to run this skill.
  • Do not read an empty run list on a fork PR as a broken or pending run. Check isCrossRepository first, then triage the offline check instead.
  • Do not declare a verdict from metadata alone when result: changed. Pull the baseline and current PNGs and look at them; metadata can only say "something changed", not whether the change is intended.