AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

expo-skill-eval

Evaluate Expo skills in this repo end-to-end - trigger accuracy, generated code quality, and runtime screenshots on iOS simulator and Android emulator via Expo Go (web optional). Use when the user wants to eval an Expo skill, test that a skill produces working code, benchmark a skill with device screenshots, or verify a skill's output renders correctly.

Original source
expo
License
MIT
Pinned revision
05754e71d02abe0d2d7a2fb5259ccac41f5ea59c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read expo--expo-skill-eval at revision 05754e71d02abe0d2d7a2fb5259ccac41f5ea59c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 4

SKILL.md · 24000

aches it under ~/.cache/expo-skill-eval/fixtures/, and clones the cache with APFS copy-on-write — so the first run per variant pays the install cost and every later run is near-instant. The default clean variant runs the template's reset-project script, so executors start from a blank app and every screen in the output is theirs — a much cleaner grading signal. Use full only when the eval prompt assumes an existing app (e.g. "I have an app with two tabs..."). The script also resets git inside the clone, so git diff in the app shows exactly what the executor changed (useful evidence for the grader).

Build fixtures sequentially, then fan out executors — never create fixtures concurrently. make-fixture.sh shares a cache under ~/.cache/expo-skill-eval/fixtures/ keyed by SDK+variant. If two runs both find the cache cold and call bunx create-expo-app at the same time, bun's link step collides and one fails with EEXIST / "could not determine executable to run for package create-expo-app". So in the executor orchestrator (step 3), create all fixtures one at a time first — a plain Python loop calling subprocess.run(["bash", "<scripts>/make-fixture.sh", app, sdk, variant]) (where sdk is the version chosen up front) — then fan out the claude -p executors with a ThreadPoolExecutor. Sequential creation is cheap: only the first fixture per SDK+variant pays the install cost; the rest are ~1s APFS clones. (And never fan fixtures out with ad-hoc shell like make-fixture.sh A & make-fixture.sh B & wait — the &/wait segments prompt; the sequential Python loop avoids both the race and the prompt.)

3. Generate (executor subagents)

Run executors as claude -p subprocess calls from a Python script, not via the Agent tool. The Agent tool spawns subagents with their own permission context — file edits inside the fixture app will prompt the user. A claude -p subprocess is a separate process outside the permission system entirely (the same pattern the trigger eval harness uses).

Write a Python script to /private/tmp/expo-skill-eval-<skill>/run_executors.py. First create every run's fixture in a sequential loop — subprocess.run(["bash", "<scripts>/make-fixture.sh", app, sdk, variant], …) one at a time (concurrent creation races the shared bun cache — see step 2). Then run the with-skill and without-skill claude -p calls in parallel via a ThreadPoolExecutor. Both phases live inside Python (covered by the python3 rule), so nothing runs as ad-hoc shell on the main thread. Each executor prompt must include:

  • The skill path (with-skill runs only) and the eval prompt.
  • Image-prompt cases (reference_image set): the absolute path to the target screenshot plus an instruction like "Open the reference screenshot at <path> with your Read tool and build an app whose UI matches it as closely as you can — layout, components, spacing, and colors." (claude -p renders PNGs read this way, so the executor can actually see the target.)
  • The fixture app path: "Make your changes inside <app-path>. The project already exists and has dependencies installed. Use absolute paths for all file operations."
  • "Before writing any files, inspect the project layout — run ls, read package.json and app.json — to find the correct routes directory. Recent SDK default templates place Expo Router routes in src/app/; older ones use app/ at the project root — inspect to confirm which this fixture uses."
  • "Do NOT start the dev server, boot simulators, or take screenshots — the harness does that after you finish."
  • Where to save a short summary of what was built.

Flags for the claude -p subprocess:

  • Strip CLAUDECODE from the environment (env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}) — otherwise claude -p hangs silently when nested inside a running Claude Code session.
  • A permission flag, confirmed with the user up front (see Before starting — clarify scope): either --dangerously-skip-permissions or --permission-mode acceptEdits. Bake the chosen flag into the generated script. A bare claude -p with neither flag can't write files — it has no TTY to approve the edit and emits code as text instead.
  • Do NOT pass --plugin-dir to executors (unlike the trigger eval). The with-skill run already reads the skill by its absolute SKILL_PATH, so it tests the local content directly; and the without-skill run must have no skill available at all — loading the plugin would let the skill auto-trigger and contaminate the baseline. Keeping executors path-based also cleanly separates the two questions: the executor measures content quality (is the skill useful once read?), the trigger eval measures triggering (does the description get it picked?).

Capture stdout/stderr per run to a log file next to the fixture for grading evidence. Set timeout to 900s per executor — with-skill runs read multiple reference files before coding and regularly take 5–10 minutes.

4. Static gate

Write run_static.py and run it with python3. For each eval/config app it calls subprocess.run(["bash", "<scripts>/check-static.sh", app, "ios,android"], capture_output=True, …) across a ThreadPoolExecutor (static gates are independent — run them concurrently inside Python, never with shell &/wait), and writes each result to eval-<i>/<config>/static.json (exit code + captured output) for the grader.

check-static.sh runs tsc --noEmit, expo lint, and expo export for the listed platforms. A passing export catches most import/syntax/missing-module failures without touching a device; a failing export short-circuits step 5 with a clean FAIL — record it and have the snapshot orchestrator skip that app.

5. Run + screenshot (serial across evals)

Write run_snapshots.py and run it with python3. Simulators and emulators are shared resources, so this orchestrator runs serially (no thread pool): for each app that passed the static gate, and each platform, it os.makedirs the outputs/ dir and calls subprocess.run(["bash", "<scripts>/snapshot-<platform>.sh", app, f"{outputs}/<platform>.png", port], env={**os.environ, "EXPO_SKILL_EVAL_RUNNER": runner}, …). Pass the port as a positional argument: use 8081 for iOS and 8082 for Android — expo run:ios/android --port N is supported and using separate ports lets you run both platforms without port collisions if you ever parallelize. Screenshots land in the run's outputs/ directory so the viewer renders them inline.

Reclaim disk after each fixture — essential for dev-build runs. Once all selected platforms for an app are captured (and before the next fixture builds), call subprocess.run(["bash", "<scripts>/clean-fixture.sh", app]). Each expo run:<platform> leaves multi-GB native build output (iOS Pods + DerivedData, Android Gradle build); without this, evals × configs × iterations pile up and fill the disk mid-run (the instability you'll see is the disk filling). clean-fixture.sh removes the heavy regenerable dirs (node_modules, ios, android, .expo, dist) and the fixture's iOS DerivedData, keeping the app source + git so the grader's git diff still works. With serial snapshots + per-fixture cleanup, peak disk stays at ~one fixture's build instead of all of them. (Harmless for expo-go runs too — they just have little to reclaim.)

runner is the up-front choice (expo-go default, or dev-build). The snapshot scripts honor EXPO_SKILL_EVAL_RUNNER: expo-go launches with expo start --<platform> (and the Expo Go install/deep-link dance); dev-build launches with expo run:<platform> --port <port>, which compiles+installs a native dev client and skips the Expo Go steps. The scripts already default the dev-build timeout to 900s, but bump EXPO_SKILL_EVAL_BUNDLE_TIMEOUT higher if the first native compile needs it. make-fixture.sh pre-installs expo-dev-client in every fixture so the dev-client URL scheme is registered before expo run tries to deep-link the app open.

Snapshot scripts always capture the initial route /. They open the app via a deep link and take one screenshot — they cannot tap or navigate. Design eval prompts so the feature under test renders at the root route. If the executor places the main UI behind a navigation action (e.g. an "Open Settings" button on the index), the snapshot will miss the feature entirely and all visual expectations will fail.

Each snapshot-<platform>.sh frees its Metro port on startup (kills any stale process left on it by a crashed prior run) and tears Metro down on exit — so you never need to run lsof/kill/pkill yourself to clear ports (that would prompt, and it's already handled). It then starts Metro, waits for the "Bundled" line in the Metro log, settles, captures a screenshot, and tears Metro down. iOS boots the newest available iPhone simulator if none is booted; Android boots the first AVD if no device is attached (the slow path — boot once and reuse across the whole iteration). Android first recycles a wedged/offline emulator (graceful adb emu kill, then force-kill + adb reset) so a half-dead instance can't poison the run, and boots with hardware GPU (-gpu host, Metal-accelerated on Apple Silicon). If host self-aborts the emulator on a given machine (qemu SIGABRT deep in gfxstream/Metal — possible on Apple Silicon under load), edit GPU_MODE in snapshot-android.sh to a software mode (guest renders reliably but slowly — bump the settle; avoid swiftshader_indirect, which hangs at boot on arm64). snapshot-web.sh runs only when platforms includes web. Each script writes a Metro log next to the screenshot (<name>.metro.log) — include it in the grader's inputs. If a script exits non-zero it still attempts a best-effort screenshot (an error screen is evidence too). dev-build relaunch: after Metro is up, the scripts relaunch the app via xcrun simctl launch (iOS) and adb shell am start -n <pkg>/.MainActivity (Android) — both avoid the "Open in X?" system dialog that a URL-scheme deep link triggers on first launch.

After all screenshots for the iteration are captured, always generate the viewer — pass the workspace root to the checked-in script:

python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill>

It writes viewer.html into the workspace root (one level above iteration-N/) and opens it in the browser itself (via webbrowser.open) — so no separate open command (and no Bash(open:*) rule) is needed. See the Viewer section below.

6. Grade

Spawn a grader subagent in the foreground. Its prompt must include:

  • The eval prompt, expectations list, and visual_expectations from the eval case.
  • The instructions in agents/visual-grader.md (screenshot grading, redbox detection).
  • The screenshot files, Metro logs, and the step-4 static.json as inputs.
  • Image-prompt cases (case has reference_image): also include the target screenshot (reference_image), references/design-rubric.md, and the fixture's git diff. Tell the grader to compare the generated screenshot(s) to the target and emit the reference_match + quality blocks below.

The grader writes grading.json next to the outputs with this shape:

{
  "score": 8.5,
  "max_score": 9,
  "expectations": [
    {"text": "...", "passed": true, "evidence": "..."}
  ],
  "reference_match": {
    "score": 7, "max": 10,
    "evidence": "ios.png vs target.png: same two-section grouped list + toggle; accent color differs (blue vs target's green); row spacing tighter than target"
  },
  "quality": {
    "dimensions": [
      {"name": "Layout & hierarchy", "score": 2, "max": 3, "evidence": "ios.png: …"}
    ],
    "subtotal": 17,
    "max": 24,
    "summary": "…"
  },
  "user_notes_summary": {"needs_review": false, "notes": ""}
}

Visual expectations go into the same expectations array with evidence naming the screenshot file and describing what is visible. The reference_match block (how closely the generated app reproduces the target screenshot) and the quality block (design-rubric scores from references/design-rubric.md) are emitted only for image-prompt cases — or when a quality grade is explicitly requested. Omit both for plain text-prompt runs.

Rollout phases

Build out and debug the pipeline in this order — each phase is independently useful:

  1. Static: steps 1–4 only (runtime.mode: "static-only" for everything). No devices needed; CI-friendly.
  2. iOS: add snapshot-ios.sh to the loop. simctl is the most scriptable target.
  3. Android: add snapshot-android.sh. Emulator boot is the slowest part — keep one emulator running for the whole session.
  4. Web: add snapshot-web.sh for skills that target web (uses Playwright via bunx; first run downloads Chromium).

Practical notes

  • Temp locations: all eval workspaces go under /private/tmp/expo-skill-eval-<skill-name>/iteration-N/. Everything in this run — Read, Write, Edit, and Bash — is covered by the allowed-tools frontmatter, so a correctly-loaded skill runs prompt-free.

  • Permission rule forms (why this skill stays prompt-free): the rule syntax matters and the two tool families behave differently:

    • Bash(...) rules — path-scoped to the skill's own code (no broad interpreters). Bash(python3 /private/tmp/expo-skill-eval-*) (plus the /tmp alias) runs the Python orchestrators you generate under the workspace; Bash(python3 *expo-skill-eval/scripts/*) runs the checked-in scripts/generate_viewer.py; Bash(tee /private/tmp/expo-skill-eval-*) (+ /tmp) lets python3 … 2>&1 | tee <workspace>/…log write a log without prompting; Bash(bash *expo-skill-eval/scripts/*) runs only this skill's scripts/*.sh. Because every path is pinned, the escape hatches stay denied: python3 -c …, bash -c …, tee /etc/…, and running code anywhere else do not match (verified empirically — a scoped rule allows bash <dir>/run.sh but blocks bash -c … and any other path). Commands the scripts call internally — bunx, xcrun simctl, adb, git, mkdir, expo — are children of the script, not Bash tool calls, so they need no rule. Do not run ad-hoc mkdir/ls/find/cat/grep from the main thread (they have no rule and prompt — and a raw mkdir "$WORKSPACE/…" can't match a path glob because the path is an unexpanded variable): create the directory tree with make-workspace.sh (step 0), let orchestrators create their own dirs (os.makedirs), and inspect results with the Read/Glob/Grep tools (no Bash rule needed).
    • Bash rule matching (tested, non-obvious): a Bash rule is a gitignore-style glob over the command string. * matches any run of characters including / and spaces and works mid-pattern — so Bash(python3 /private/tmp/expo-skill-eval-*) matches python3 /private/tmp/expo-skill-eval-x/run.py 2>&1, and Bash(bash *expo-skill-eval/scripts/*) matches bash /any/abs/path/expo-skill-eval/scripts/foo.sh args. Two gotchas that burned earlier attempts: ** is matched literally (never use it in a Bash rule), and the :* suffix only works right after the command token (Bash(python3:*)) — not after a partial path (Bash(python3 /path-:*) does not match). Compound commands split on |, &&, ||, ;, & and each segment needs its own matching rule.
    • Read rules suppress prompts; Write/Edit rules do not. This is a Claude Code asymmetry (not a pattern bug, and not reload — in a session where the Bash/Read rules from this same frontmatter are clearly working, Write still prompts): file creation/editing always goes through Claude Code's edit-approval flow regardless of allowed-tools. The frontmatter still scopes Read/Write/Edit to …/expo-skill-eval-*/** (both the /tmp and /private/tmp forms, since macOS doesn't auto-resolve the symlink) as documentation and a guardrail, but those Write/Edit entries won't silence the prompt on their own. Practical consequence: at the start of a run you get one Write prompt for the workspace — choose "Yes, allow all edits in this directory for the session" and every later orchestrator / evals.json / viewer write under that workspace goes through silently. That single directory approval, not a rule, is what makes file-writing prompt-free.
    • Reload after editing frontmatter — a full restart, not /reload-skills. allowed-tools is read once when the skill loads at session start; /reload-skills reloads the skill body but does not reliably refresh the permission rules. After editing this file, quit Claude Code entirely and start a new session, then re-run the skill — otherwise a stale (cached) ruleset keeps prompting even though the file on disk is correct.
    • Grader subagents run with their own permission context and will still prompt for file access — that is expected and separate from the main thread's rules.
  • Calling eval scripts — one standalone command, never chained. Invoke each script as its own Bash call with an absolute path: bash /abs/path/expo-skill-eval/scripts/snapshot-ios.sh arg1 arg2 (covered by Bash(bash *expo-skill-eval/scripts/*)). Do not combine it with &, &&, ||, ;, wait, tail, head, or echo — compound commands are checked per segment, and those extra segments have no rule, so the whole thing prompts even though the bash …/scripts/… part is allowed. (The one allowed pipe is … 2>&1 | tee <workspace>/…log, since the scoped tee rule covers it.) Need parallelism or output trimming? Put it in a Python orchestrator (covered by python3 /…/expo-skill-eval-*), which runs scripts via subprocess across a ThreadPoolExecutor. Inspect results with the Read/Glob/Grep tools, not cat/ls/grep. General rule: under this skill's tight scoping, any ad-hoc shell the agent improvises will prompt — the fix is to move it into a script/orchestrator (or use the scoped tee), never to broaden a rule.

  • Inspecting outputs (screenshots, logs, files) — use tools, not shell. To find files use the Glob tool (e.g. /private/tmp/expo-skill-eval-<skill>/iteration-N/**/ios.png); to view them use the Read tool — Read renders PNGs visually, which is exactly what you need to confirm a screenshot rendered. To search file contents use Grep. Never use find/ls/cat for this: they prompt, and find … -exec … is deliberately not allowed because its -exec can run anything (e.g. -exec rm). These tools are scoped and prompt-free; reach for them every time you'd otherwise type find/ls/cat.

  • Generated Python scripts: write orchestration/aggregation scripts under the workspace (e.g. /private/tmp/expo-skill-eval-<skill>/aggregate.py) and run them with python3 (covered by Bash(python3 /private/tmp/expo-skill-eval-*)). The viewer is the exception — it's the checked-in scripts/generate_viewer.py, run via Bash(python3 *expo-skill-eval/scripts/*). Write auto-creates parent dirs but prompts the first time — approve the workspace directory once (see the Write/Edit note above). Capture output either by having the script write its own log or via python3 … 2>&1 | tee <workspace>/…log (covered by the scoped tee rule); read logs back with the Read tool. Don't use python3 -c … for setup (the scoped rule only matches a workspace script path, so a bare -c prompts).

  • Trigger evals vs installed plugin: detect the real installed skill name (e.g. expo:expo-ui) in the stream — a synthetic-duplicate harness always scores 0% when the real plugin is installed because the model picks the genuine skill over the synthetic copy.

  • Benchmark aggregation: save each run's grading.json + timing.json under eval-<N>/<config>/run-1/. Write a Python aggregation script under the workspace and run it with python3.

  • Expo Go ceiling: anything requiring custom native code (expo-module, App Clips, brownfield) cannot run in Expo Go. Use static-only mode for those — see references/runtime-matrix.md before writing eval cases for a skill (note: @expo/ui does run in Expo Go on SDK 56+).

  • API-route skills: instead of a screenshot, verify with curl against the route while Metro is up; record the response as an output file for grading.

  • Timing data: capture token counts and duration into timing.json immediately after each executor run — it is not recoverable later. To capture token counts, add --output-format=stream-json --verbose to the executor claude -p call and parse the message_start / message_delta events from the log. Without these flags the log only contains prose and elapsed seconds are the only recoverable metric.

  • First-launch dialogs: Expo Go occasionally shows a one-time prompt on a fresh simulator. If a screenshot captures a dialog instead of the app, re-run the snapshot script (it reopens the URL) and re-capture.

Viewer

After taking screenshots, always generate and open the HTML viewer so the user can see results immediately without being asked. The viewer is the checked-in scripts/generate_viewer.py — run it with the workspace root as its argument:

python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill>

It writes a self-contained /private/tmp/expo-skill-eval-<skill>/viewer.html and opens it in the browser itself (webbrowser.open). What it renders:

  • A tab per iteration (iteration-* under the workspace root; remembers the last active tab in localStorage).
  • For each eval case (read from <iteration>/evals.json): side-by-side with_skill / without_skill columns, each showing static-gate status, score, the platform screenshots (click to zoom; embedded as base64 data: URIs so the file is self-contained), the expectation list with PASS/FAIL badges, and reviewer notes.
  • For image-prompt cases (a grading.json with reference_match / quality): the target screenshot beside the generated ones, the reference_match score (generated vs target), the quality rubric per config (one bar per dimension with its score/max plus the subtotal), and the quality delta (with_skill − without_skill subtotal) in the summary bar alongside the correctness delta.
  • A summary bar with with_skill %, without_skill %, and delta.
  • A trigger accuracy table when trigger-evals/trigger_results.json exists.
  • A dark background with color-coded scores (green ≥85%, amber ≥65%, red below).

Publishing the viewer (only if opted in up front)

The local viewer.html is always generated. Only when the user chose "Publish a shareable Artifact" in the up-front confirmation, additionally render it to a claude.ai Artifact at the very end — never publish without that opt-in (it's outward-facing and a published page can be cached/indexed). Mechanics:

  • The Artifact tool wraps the file in its own <!doctype html>…<head></head><body> skeleton, so the file you hand it must be page content only — inline <style>/<script>, base64 data: images, and a <title>, but no <!DOCTYPE>/<html>/<head>/<body> tags of its own (a full standalone document gets double-wrapped and renders wrong).
  • The script emits an Artifact-friendly variant when you add --artifact: python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill> --artifact writes viewer_artifact.html (same content, skeleton stripped, no browser open). Pass that file to the Artifact tool (favicon: "📊"), not the standalone one.
  • The viewer is already self-contained (base64 screenshots, inline CSS/JS), so it satisfies the Artifact CSP (no external hosts).

References

  • references/runtime-matrix.md — per-skill runtime applicability (expo-go vs static-only, platform notes).
  • `agents/vis