AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

grade-iterate

Use when a launched managed agent needs grading against a rubric and improving ('grade my agent', 'is it good enough'). Builds the outcome with outcome_builder.py, reads verdicts with verdict_reader.py and runs held-back evals from eval_scaffold.py.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--grade-iterate at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 2

SKILL.md

Phase 3 — Grade → Iterate (the bounded loop)

This is the plugin's loop: CMA's outcome primitive self-grades the agent's work in an isolated context and feeds failing verdicts back for the next attempt. It is always bounded by max_iterations (1..20) — never "improve forever".

See ../../references/loops-and-workflows.md and the outcome section of ../../references/cma-primitives.md.

Workflow

  1. Define the outcome.
    python3 scripts/outcome_builder.py \
      --sheet ./my-agent/build-sheet.json --max-iterations 5 \
      --out ./my-agent/payloads/outcome.json
    
    The rubric is required; max_iterations is clamped to 1..20. Send the payload as a user.define_outcome event (append to the running session).
  2. Read every verdict first.
    python3 scripts/verdict_reader.py --result ./my-agent/last-verdict.json
    
    Tables the rubric outcome and recommends: SHIP (satisfied), SHARPEN then re-run (needs_revision), ESCALATE (max_iterations_reached / failed), RESUME (interrupted). With ≤1 iteration left it flips to "make the single highest-value fix or escalate now".
  3. Loop invariant. Each iteration must move ≥1 rubric line fail→pass, or the run halts at the cap and escalates. Don't burn the budget on cosmetic edits.
  4. Once a version passes, run held-back eval.
    python3 scripts/eval_scaffold.py \
      --sheet ./my-agent/build-sheet.json --out ./my-agent/eval.json --concurrency 5
    
    Held-back cases (never seen during iteration) run in parallel, capped at the 25-thread CMA ceiling, each graded against the same rubric.
  5. Decide. SHIP as v0, or promote to a scheduled deployment (Phase 4). Record the verdict on the goal: goal_state.py set --phase run-without-you.

Hard rules

  • Bounded, always. No outcome without a max_iterations cap.
  • Read the verdict before acting. The grader's explanation drives the next move.
  • Held-back cases are held back. Never grade generalization on cases the agent already iterated against.

Forcing-question library (recommend + cite)

  1. "What are the 3–5 rubric lines?" Recommend: grounded, checkable criteria. Cite: cma-primitives.md (rubric required).
  2. "How many iterations before you'd rather look yourself?" Recommend: 3–5. Cite: loops-and-workflows.md (bounded loop).
  3. "On a fail, sharpen the prompt or the tools?" Recommend: whichever rubric line failed points to. Cite: verdict_reader next-move table.
  4. "Which cases did the agent NOT see?" Recommend: hold back ≥3 for generalization. Cite: this SKILL (held-back eval).

Tools

  • scripts/outcome_builder.py — user.define_outcome payload (rubric required, cap 1..20).
  • scripts/verdict_reader.py — grader result → next move.
  • scripts/eval_scaffold.py — held-back cases + parallel run plan (≤25 threads).