SKILL.md · 24000
aches it under ~/.cache/expo-skill-eval/fixtures/, and clones the cache with APFS copy-on-write — so the first run per variant pays the install cost and every later run is near-instant. The default clean variant runs the template's reset-project script, so executors start from a blank app and every screen in the output is theirs — a much cleaner grading signal. Use full only when the eval prompt assumes an existing app (e.g. "I have an app with two tabs..."). The script also resets git inside the clone, so git diff in the app shows exactly what the executor changed (useful evidence for the grader).
Build fixtures sequentially, then fan out executors — never create fixtures concurrently. make-fixture.sh shares a cache under ~/.cache/expo-skill-eval/fixtures/ keyed by SDK+variant. If two runs both find the cache cold and call bunx create-expo-app at the same time, bun's link step collides and one fails with EEXIST / "could not determine executable to run for package create-expo-app". So in the executor orchestrator (step 3), create all fixtures one at a time first — a plain Python loop calling subprocess.run(["bash", "<scripts>/make-fixture.sh", app, sdk, variant]) (where sdk is the version chosen up front) — then fan out the claude -p executors with a ThreadPoolExecutor. Sequential creation is cheap: only the first fixture per SDK+variant pays the install cost; the rest are ~1s APFS clones. (And never fan fixtures out with ad-hoc shell like make-fixture.sh A & make-fixture.sh B & wait — the &/wait segments prompt; the sequential Python loop avoids both the race and the prompt.)
3. Generate (executor subagents)
Run executors as claude -p subprocess calls from a Python script, not via the Agent tool. The Agent tool spawns subagents with their own permission context — file edits inside the fixture app will prompt the user. A claude -p subprocess is a separate process outside the permission system entirely (the same pattern the trigger eval harness uses).
Write a Python script to /private/tmp/expo-skill-eval-<skill>/run_executors.py. First create every run's fixture in a sequential loop — subprocess.run(["bash", "<scripts>/make-fixture.sh", app, sdk, variant], …) one at a time (concurrent creation races the shared bun cache — see step 2). Then run the with-skill and without-skill claude -p calls in parallel via a ThreadPoolExecutor. Both phases live inside Python (covered by the python3 rule), so nothing runs as ad-hoc shell on the main thread. Each executor prompt must include:
- The skill path (with-skill runs only) and the eval prompt.
- Image-prompt cases (
reference_imageset): the absolute path to the target screenshot plus an instruction like "Open the reference screenshot at<path>with your Read tool and build an app whose UI matches it as closely as you can — layout, components, spacing, and colors." (claude -prenders PNGs read this way, so the executor can actually see the target.) - The fixture app path: "Make your changes inside
<app-path>. The project already exists and has dependencies installed. Use absolute paths for all file operations." - "Before writing any files, inspect the project layout — run
ls, readpackage.jsonandapp.json— to find the correct routes directory. Recent SDK default templates place Expo Router routes insrc/app/; older ones useapp/at the project root — inspect to confirm which this fixture uses." - "Do NOT start the dev server, boot simulators, or take screenshots — the harness does that after you finish."
- Where to save a short summary of what was built.
Flags for the claude -p subprocess:
- Strip
CLAUDECODEfrom the environment (env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}) — otherwiseclaude -phangs silently when nested inside a running Claude Code session. - A permission flag, confirmed with the user up front (see Before starting — clarify scope): either
--dangerously-skip-permissionsor--permission-mode acceptEdits. Bake the chosen flag into the generated script. A bareclaude -pwith neither flag can't write files — it has no TTY to approve the edit and emits code as text instead. - Do NOT pass
--plugin-dirto executors (unlike the trigger eval). The with-skill run already reads the skill by its absoluteSKILL_PATH, so it tests the local content directly; and the without-skill run must have no skill available at all — loading the plugin would let the skill auto-trigger and contaminate the baseline. Keeping executors path-based also cleanly separates the two questions: the executor measures content quality (is the skill useful once read?), the trigger eval measures triggering (does the description get it picked?).
Capture stdout/stderr per run to a log file next to the fixture for grading evidence. Set timeout to 900s per executor — with-skill runs read multiple reference files before coding and regularly take 5–10 minutes.
4. Static gate
Write run_static.py and run it with python3. For each eval/config app it calls subprocess.run(["bash", "<scripts>/check-static.sh", app, "ios,android"], capture_output=True, …) across a ThreadPoolExecutor (static gates are independent — run them concurrently inside Python, never with shell &/wait), and writes each result to eval-<i>/<config>/static.json (exit code + captured output) for the grader.
check-static.sh runs tsc --noEmit, expo lint, and expo export for the listed platforms. A passing export catches most import/syntax/missing-module failures without touching a device; a failing export short-circuits step 5 with a clean FAIL — record it and have the snapshot orchestrator skip that app.
5. Run + screenshot (serial across evals)
Write run_snapshots.py and run it with python3. Simulators and emulators are shared resources, so this orchestrator runs serially (no thread pool): for each app that passed the static gate, and each platform, it os.makedirs the outputs/ dir and calls subprocess.run(["bash", "<scripts>/snapshot-<platform>.sh", app, f"{outputs}/<platform>.png", port], env={**os.environ, "EXPO_SKILL_EVAL_RUNNER": runner}, …). Pass the port as a positional argument: use 8081 for iOS and 8082 for Android — expo run:ios/android --port N is supported and using separate ports lets you run both platforms without port collisions if you ever parallelize. Screenshots land in the run's outputs/ directory so the viewer renders them inline.
Reclaim disk after each fixture — essential for dev-build runs. Once all selected platforms for an app are captured (and before the next fixture builds), call subprocess.run(["bash", "<scripts>/clean-fixture.sh", app]). Each expo run:<platform> leaves multi-GB native build output (iOS Pods + DerivedData, Android Gradle build); without this, evals × configs × iterations pile up and fill the disk mid-run (the instability you'll see is the disk filling). clean-fixture.sh removes the heavy regenerable dirs (node_modules, ios, android, .expo, dist) and the fixture's iOS DerivedData, keeping the app source + git so the grader's git diff still works. With serial snapshots + per-fixture cleanup, peak disk stays at ~one fixture's build instead of all of them. (Harmless for expo-go runs too — they just have little to reclaim.)
runner is the up-front choice (expo-go default, or dev-build). The snapshot scripts honor EXPO_SKILL_EVAL_RUNNER: expo-go launches with expo start --<platform> (and the Expo Go install/deep-link dance); dev-build launches with expo run:<platform> --port <port>, which compiles+installs a native dev client and skips the Expo Go steps. The scripts already default the dev-build timeout to 900s, but bump EXPO_SKILL_EVAL_BUNDLE_TIMEOUT higher if the first native compile needs it. make-fixture.sh pre-installs expo-dev-client in every fixture so the dev-client URL scheme is registered before expo run tries to deep-link the app open.
Snapshot scripts always capture the initial route /. They open the app via a deep link and take one screenshot — they cannot tap or navigate. Design eval prompts so the feature under test renders at the root route. If the executor places the main UI behind a navigation action (e.g. an "Open Settings" button on the index), the snapshot will miss the feature entirely and all visual expectations will fail.
Each snapshot-<platform>.sh frees its Metro port on startup (kills any stale process left on it by a crashed prior run) and tears Metro down on exit — so you never need to run lsof/kill/pkill yourself to clear ports (that would prompt, and it's already handled). It then starts Metro, waits for the "Bundled" line in the Metro log, settles, captures a screenshot, and tears Metro down. iOS boots the newest available iPhone simulator if none is booted; Android boots the first AVD if no device is attached (the slow path — boot once and reuse across the whole iteration). Android first recycles a wedged/offline emulator (graceful adb emu kill, then force-kill + adb reset) so a half-dead instance can't poison the run, and boots with hardware GPU (-gpu host, Metal-accelerated on Apple Silicon). If host self-aborts the emulator on a given machine (qemu SIGABRT deep in gfxstream/Metal — possible on Apple Silicon under load), edit GPU_MODE in snapshot-android.sh to a software mode (guest renders reliably but slowly — bump the settle; avoid swiftshader_indirect, which hangs at boot on arm64). snapshot-web.sh runs only when platforms includes web. Each script writes a Metro log next to the screenshot (<name>.metro.log) — include it in the grader's inputs. If a script exits non-zero it still attempts a best-effort screenshot (an error screen is evidence too). dev-build relaunch: after Metro is up, the scripts relaunch the app via xcrun simctl launch (iOS) and adb shell am start -n <pkg>/.MainActivity (Android) — both avoid the "Open in X?" system dialog that a URL-scheme deep link triggers on first launch.
After all screenshots for the iteration are captured, always generate the viewer — pass the workspace root to the checked-in script:
python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill>
It writes viewer.html into the workspace root (one level above iteration-N/) and opens it in the browser itself (via webbrowser.open) — so no separate open command (and no Bash(open:*) rule) is needed. See the Viewer section below.
6. Grade
Spawn a grader subagent in the foreground. Its prompt must include:
- The eval prompt, expectations list, and visual_expectations from the eval case.
- The instructions in
agents/visual-grader.md(screenshot grading, redbox detection). - The screenshot files, Metro logs, and the step-4
static.jsonas inputs. - Image-prompt cases (case has
reference_image): also include the target screenshot (reference_image),references/design-rubric.md, and the fixture'sgit diff. Tell the grader to compare the generated screenshot(s) to the target and emit thereference_match+qualityblocks below.
The grader writes grading.json next to the outputs with this shape:
{
"score": 8.5,
"max_score": 9,
"expectations": [
{"text": "...", "passed": true, "evidence": "..."}
],
"reference_match": {
"score": 7, "max": 10,
"evidence": "ios.png vs target.png: same two-section grouped list + toggle; accent color differs (blue vs target's green); row spacing tighter than target"
},
"quality": {
"dimensions": [
{"name": "Layout & hierarchy", "score": 2, "max": 3, "evidence": "ios.png: …"}
],
"subtotal": 17,
"max": 24,
"summary": "…"
},
"user_notes_summary": {"needs_review": false, "notes": ""}
}
Visual expectations go into the same expectations array with evidence naming the screenshot file and describing what is visible. The reference_match block (how closely the generated app reproduces the target screenshot) and the quality block (design-rubric scores from references/design-rubric.md) are emitted only for image-prompt cases — or when a quality grade is explicitly requested. Omit both for plain text-prompt runs.
Rollout phases
Build out and debug the pipeline in this order — each phase is independently useful:
- Static: steps 1–4 only (
runtime.mode: "static-only"for everything). No devices needed; CI-friendly. - iOS: add
snapshot-ios.shto the loop.simctlis the most scriptable target. - Android: add
snapshot-android.sh. Emulator boot is the slowest part — keep one emulator running for the whole session. - Web: add
snapshot-web.shfor skills that target web (uses Playwright viabunx; first run downloads Chromium).
Practical notes
-
Temp locations: all eval workspaces go under
/private/tmp/expo-skill-eval-<skill-name>/iteration-N/. Everything in this run —Read,Write,Edit, andBash— is covered by theallowed-toolsfrontmatter, so a correctly-loaded skill runs prompt-free. -
Permission rule forms (why this skill stays prompt-free): the rule syntax matters and the two tool families behave differently:
Bash(...)rules — path-scoped to the skill's own code (no broad interpreters).Bash(python3 /private/tmp/expo-skill-eval-*)(plus the/tmpalias) runs the Python orchestrators you generate under the workspace;Bash(python3 *expo-skill-eval/scripts/*)runs the checked-inscripts/generate_viewer.py;Bash(tee /private/tmp/expo-skill-eval-*)(+/tmp) letspython3 … 2>&1 | tee <workspace>/…logwrite a log without prompting;Bash(bash *expo-skill-eval/scripts/*)runs only this skill'sscripts/*.sh. Because every path is pinned, the escape hatches stay denied:python3 -c …,bash -c …,tee /etc/…, and running code anywhere else do not match (verified empirically — a scoped rule allowsbash <dir>/run.shbut blocksbash -c …and any other path). Commands the scripts call internally —bunx,xcrun simctl,adb,git,mkdir,expo— are children of the script, not Bash tool calls, so they need no rule. Do not run ad-hocmkdir/ls/find/cat/grepfrom the main thread (they have no rule and prompt — and a rawmkdir "$WORKSPACE/…"can't match a path glob because the path is an unexpanded variable): create the directory tree withmake-workspace.sh(step 0), let orchestrators create their own dirs (os.makedirs), and inspect results with theRead/Glob/Greptools (no Bash rule needed).- Bash rule matching (tested, non-obvious): a Bash rule is a gitignore-style glob over the command string.
*matches any run of characters including/and spaces and works mid-pattern — soBash(python3 /private/tmp/expo-skill-eval-*)matchespython3 /private/tmp/expo-skill-eval-x/run.py 2>&1, andBash(bash *expo-skill-eval/scripts/*)matchesbash /any/abs/path/expo-skill-eval/scripts/foo.sh args. Two gotchas that burned earlier attempts:**is matched literally (never use it in a Bash rule), and the:*suffix only works right after the command token (Bash(python3:*)) — not after a partial path (Bash(python3 /path-:*)does not match). Compound commands split on|,&&,||,;,&and each segment needs its own matching rule. Readrules suppress prompts;Write/Editrules do not. This is a Claude Code asymmetry (not a pattern bug, and not reload — in a session where theBash/Readrules from this same frontmatter are clearly working,Writestill prompts): file creation/editing always goes through Claude Code's edit-approval flow regardless ofallowed-tools. The frontmatter still scopesRead/Write/Editto…/expo-skill-eval-*/**(both the/tmpand/private/tmpforms, since macOS doesn't auto-resolve the symlink) as documentation and a guardrail, but thoseWrite/Editentries won't silence the prompt on their own. Practical consequence: at the start of a run you get one Write prompt for the workspace — choose "Yes, allow all edits in this directory for the session" and every later orchestrator /evals.json/ viewer write under that workspace goes through silently. That single directory approval, not a rule, is what makes file-writing prompt-free.- Reload after editing frontmatter — a full restart, not
/reload-skills.allowed-toolsis read once when the skill loads at session start;/reload-skillsreloads the skill body but does not reliably refresh the permission rules. After editing this file, quit Claude Code entirely and start a new session, then re-run the skill — otherwise a stale (cached) ruleset keeps prompting even though the file on disk is correct. - Grader subagents run with their own permission context and will still prompt for file access — that is expected and separate from the main thread's rules.
-
Calling eval scripts — one standalone command, never chained. Invoke each script as its own Bash call with an absolute path:
bash /abs/path/expo-skill-eval/scripts/snapshot-ios.sh arg1 arg2(covered byBash(bash *expo-skill-eval/scripts/*)). Do not combine it with&,&&,||,;,wait,tail,head, orecho— compound commands are checked per segment, and those extra segments have no rule, so the whole thing prompts even though thebash …/scripts/…part is allowed. (The one allowed pipe is… 2>&1 | tee <workspace>/…log, since the scopedteerule covers it.) Need parallelism or output trimming? Put it in a Python orchestrator (covered bypython3 /…/expo-skill-eval-*), which runs scripts viasubprocessacross aThreadPoolExecutor. Inspect results with theRead/Glob/Greptools, notcat/ls/grep. General rule: under this skill's tight scoping, any ad-hoc shell the agent improvises will prompt — the fix is to move it into a script/orchestrator (or use the scopedtee), never to broaden a rule. -
Inspecting outputs (screenshots, logs, files) — use tools, not shell. To find files use the Glob tool (e.g.
/private/tmp/expo-skill-eval-<skill>/iteration-N/**/ios.png); to view them use the Read tool — Read renders PNGs visually, which is exactly what you need to confirm a screenshot rendered. To search file contents use Grep. Never usefind/ls/catfor this: they prompt, andfind … -exec …is deliberately not allowed because its-execcan run anything (e.g.-exec rm). These tools are scoped and prompt-free; reach for them every time you'd otherwise typefind/ls/cat. -
Generated Python scripts: write orchestration/aggregation scripts under the workspace (e.g.
/private/tmp/expo-skill-eval-<skill>/aggregate.py) and run them withpython3(covered byBash(python3 /private/tmp/expo-skill-eval-*)). The viewer is the exception — it's the checked-inscripts/generate_viewer.py, run viaBash(python3 *expo-skill-eval/scripts/*).Writeauto-creates parent dirs but prompts the first time — approve the workspace directory once (see theWrite/Editnote above). Capture output either by having the script write its own log or viapython3 … 2>&1 | tee <workspace>/…log(covered by the scopedteerule); read logs back with theReadtool. Don't usepython3 -c …for setup (the scoped rule only matches a workspace script path, so a bare-cprompts). -
Trigger evals vs installed plugin: detect the real installed skill name (e.g.
expo:expo-ui) in the stream — a synthetic-duplicate harness always scores 0% when the real plugin is installed because the model picks the genuine skill over the synthetic copy. -
Benchmark aggregation: save each run's
grading.json+timing.jsonundereval-<N>/<config>/run-1/. Write a Python aggregation script under the workspace and run it withpython3. -
Expo Go ceiling: anything requiring custom native code (expo-module, App Clips, brownfield) cannot run in Expo Go. Use
static-onlymode for those — seereferences/runtime-matrix.mdbefore writing eval cases for a skill (note:@expo/uidoes run in Expo Go on SDK 56+). -
API-route skills: instead of a screenshot, verify with
curlagainst the route while Metro is up; record the response as an output file for grading. -
Timing data: capture token counts and duration into
timing.jsonimmediately after each executor run — it is not recoverable later. To capture token counts, add--output-format=stream-json --verboseto the executorclaude -pcall and parse themessage_start/message_deltaevents from the log. Without these flags the log only contains prose and elapsed seconds are the only recoverable metric. -
First-launch dialogs: Expo Go occasionally shows a one-time prompt on a fresh simulator. If a screenshot captures a dialog instead of the app, re-run the snapshot script (it reopens the URL) and re-capture.
Viewer
After taking screenshots, always generate and open the HTML viewer so the user can see results immediately without being asked. The viewer is the checked-in scripts/generate_viewer.py — run it with the workspace root as its argument:
python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill>
It writes a self-contained /private/tmp/expo-skill-eval-<skill>/viewer.html and opens it in the browser itself (webbrowser.open). What it renders:
- A tab per iteration (
iteration-*under the workspace root; remembers the last active tab inlocalStorage). - For each eval case (read from
<iteration>/evals.json): side-by-side with_skill / without_skill columns, each showing static-gate status, score, the platform screenshots (click to zoom; embedded as base64data:URIs so the file is self-contained), the expectation list with PASS/FAIL badges, and reviewer notes. - For image-prompt cases (a
grading.jsonwithreference_match/quality): the target screenshot beside the generated ones, thereference_matchscore (generated vs target), thequalityrubric per config (one bar per dimension with its score/max plus the subtotal), and the quality delta (with_skill − without_skill subtotal) in the summary bar alongside the correctness delta. - A summary bar with with_skill %, without_skill %, and delta.
- A trigger accuracy table when
trigger-evals/trigger_results.jsonexists. - A dark background with color-coded scores (green ≥85%, amber ≥65%, red below).
Publishing the viewer (only if opted in up front)
The local viewer.html is always generated. Only when the user chose "Publish a shareable Artifact" in the up-front confirmation, additionally render it to a claude.ai Artifact at the very end — never publish without that opt-in (it's outward-facing and a published page can be cached/indexed). Mechanics:
- The
Artifacttool wraps the file in its own<!doctype html>…<head></head><body>skeleton, so the file you hand it must be page content only — inline<style>/<script>, base64data:images, and a<title>, but no<!DOCTYPE>/<html>/<head>/<body>tags of its own (a full standalone document gets double-wrapped and renders wrong). - The script emits an Artifact-friendly variant when you add
--artifact:python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill> --artifactwritesviewer_artifact.html(same content, skeleton stripped, no browser open). Pass that file to theArtifacttool (favicon: "📊"), not the standalone one. - The viewer is already self-contained (base64 screenshots, inline CSS/JS), so it satisfies the Artifact CSP (no external hosts).
References
references/runtime-matrix.md— per-skill runtime applicability (expo-go vs static-only, platform notes).- `agents/vis