AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

skill-creator

Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.

Original source
anthropics
License
Apache-2.0
Pinned revision
ca45fc496a3436251f191354091d557da31c15b5

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read anthropics--skill-creator at revision ca45fc496a3436251f191354091d557da31c15b5, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 6

SKILL.md · 24000

PTION_PLACEHOLDER__→ the skill's current description 3. Write to a temp file (e.g.,/tmp/eval_review_.html) and open it: open /tmp/eval_review_.html4. The user can edit queries, toggle should-trigger, add/remove entries, then click "Export Eval Set" 5. The file downloads to~/Downloads/eval_set.json— check the Downloads folder for the most recent version in case there are multiple (e.g.,eval_set (1).json`)

This step matters — bad eval queries lead to bad descriptions.

Step 3: Run the optimization loop

Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."

Save the eval set to the workspace, then run in the background:

python -m scripts.run_loop \
  --eval-set <path-to-trigger-eval.json> \
  --skill-path <path-to-skill> \
  --model <model-id-powering-this-session> \
  --max-iterations 5 \
  --verbose

Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.

While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.

This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with best_description — selected by test score rather than train score to avoid overfitting.

How skill triggering works

Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's available_skills list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.

This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality.

Step 4: Apply the result

Take best_description from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.


Package and Present (only if present_files tool is available)

Check whether you have access to the present_files tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:

python -m scripts.package_skill <path/to/skill-folder>

After packaging, direct the user to the resulting .skill file path so they can install it.


Claude.ai-specific instructions

In Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:

Running test cases: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.

Reviewing results: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: "How does this look? Anything you'd change?"

Benchmarking: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.

The iteration loop: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.

Description optimization: This section requires the claude CLI tool (specifically claude -p) which is only available in Claude Code. Skip it if you're on Claude.ai.

Blind comparison: Requires subagents. Skip it.

Packaging: The package_skill.py script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting .skill file.

Updating an existing skill: The user might be asking you to update an existing skill, not create a new one. In this case:

  • Preserve the original name. Note the skill's directory name and name frontmatter field -- use them unchanged. E.g., if the installed skill is research-helper, output research-helper.skill (not research-helper-v2).
  • Copy to a writeable location before editing. The installed skill path may be read-only. Copy to /tmp/skill-name/, edit there, and package from the copy.
  • If packaging manually, stage in /tmp/ first, then copy to the output directory -- direct writes may fail due to permissions.

Cowork-Specific Instructions

If you're in Cowork, the main things to know are:

  • You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)
  • You don't have a browser or display, so when generating the eval viewer, use --static <output_path> to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.
  • For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using generate_review.py (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER BEFORE evaluating inputs yourself. You want to get them in front of the human ASAP!
  • Feedback works differently: since there's no running server, the viewer's "Submit All Reviews" button will download feedback.json as a file. You can then read it from there (you may have to request access first).
  • Packaging works — package_skill.py just needs Python and a filesystem.
  • Description optimization (run_loop.py / run_eval.py) should work in Cowork just fine since it uses claude -p via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.
  • Updating an existing skill: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.

Reference files

The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.

  • agents/grader.md — How to evaluate assertions against outputs
  • agents/comparator.md — How to do blind A/B comparison between two outputs
  • agents/analyzer.md — How to analyze why one version beat another

The references/ directory has additional documentation:

  • references/schemas.md — JSON structures for evals.json, grading.json, etc.

Repeating one more time the core loop here for emphasis:

  • Figure out what the skill is about
  • Draft or edit the skill
  • Run claude-with-access-to-the-skill on test prompts
  • With the user, evaluate the outputs:
    • Create benchmark.json and run eval-viewer/generate_review.py to help the user review them
    • Run quantitative evals
  • Repeat until you and the user are satisfied
  • Package the final skill and return it to the user.

Please add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put "Create evals JSON and run eval-viewer/generate_review.py so human can review test cases" in your TodoList to make sure it happens.

Good luck!