AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

eval-driven-dev

Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read github--eval-driven-dev at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 17

references/5-run-tests.md

Step 5: Run pixie test and Fix Mechanical Issues

Why this step: Run pixie test and fix mechanical issues in your QA components — dataset format problems, runnable implementation bugs, and custom evaluator errors — until every entry produces real scores. This step is NOT about assessing result quality or fixing the application itself.


5a. Run tests

uv run pixie test

For verbose output with per-case scores and evaluator reasoning:

uv run pixie test -v

pixie test automatically loads the .env file before running tests.

The evaluation harness:

  1. Resolves the Runnable class from the dataset's runnable field
  2. Calls Runnable.create() to construct an instance, then setup() once
  3. Runs all dataset entries concurrently (up to 4 in parallel): a. Reads input_data and eval_input from the entry b. Populates the wrap input registry with eval_input data c. Initialises the capture registry d. Validates input_data into the Pydantic model and calls Runnable.run(args) e. wrap(purpose="input") calls in the app return registry values instead of calling external services f. wrap(purpose="output"/"state") calls capture data for evaluation g. Builds Evaluable from captured data h. Runs evaluators
  4. Calls Runnable.teardown() once

Because entries run concurrently, the Runnable's run() method must be concurrency-safe. If you see sqlite3.OperationalError, "database is locked", or similar errors, add a Semaphore(1) to your Runnable (see the concurrency section in Step 2 reference).

5b. Fix mechanical issues only

This step is strictly about fixing what you built in previous steps — the dataset, the runnable, and any custom evaluators. You are fixing mechanical problems that prevent the pipeline from running, NOT assessing or improving the application's output quality.

What counts as a mechanical issue (fix these):

| Error | Cause | Fix | | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | WrapRegistryMissError: name='<key>' | Dataset entry missing an eval_input item with the name that the app's wrap(purpose="input", name="<key>") expects | Add the missing {"name": "<key>", "value": ...} to eval_input in every affected entry | | WrapTypeMismatchError | Deserialized type doesn't match what the app expects | Fix the value in the dataset | | Runnable resolution failure | runnable path or class name is wrong, or the class doesn't implement the Runnable protocol | Fix filepath:ClassName in the dataset; ensure the class has create() and run() methods | | Import error | Module path or syntax error in runnable/evaluator | Fix the referenced file | | ModuleNotFoundError: pixie_qa | pixie_qa/ directory missing __init__.py | Run pixie init to recreate it | | TypeError: ... is not callable | Evaluator name points to a non-callable attribute | Evaluators must be functions, classes, or callable instances | | sqlite3.OperationalError | Concurrent run() calls sharing a SQLite connection | Add asyncio.Semaphore(1) to the Runnable (see Step 2 concurrency section) | | Custom evaluator crashes | Bug in your custom evaluator implementation | Fix the evaluator code |

What is NOT a mechanical issue (do NOT fix these here):

  • Application produces wrong/low-quality output → that's the application's behavior, analyzed in Step 6
  • Evaluator scores are low → that's a quality signal, analyzed in Step 6
  • LLM calls fail inside the application → report in Step 6, do not mock or work around
  • Evaluator scores fluctuate between runs → normal LLM non-determinism, not a bug

Iterate — fix errors, re-run, fix the next error — until pixie test runs to completion with real evaluator scores for all entries.

Output

After pixie test completes successfully, results are stored in the per-entry directory structure:

{PIXIE_ROOT}/results/<test_id>/
  meta.json                           # test run metadata
  dataset-{idx}/
    metadata.json                     # dataset name, path, runnable
    entry-{idx}/
      config.json                     # evaluators, description, expectation
      eval-input.jsonl                # input data fed to evaluators
      eval-output.jsonl               # output data captured from app
      evaluations.jsonl               # evaluation results (scored + pending)
      trace.jsonl                     # LLM call traces (if captured)

The <test_id> is printed in console output. You will reference this directory in Step 6.


If you hit an unexpected error when running tests (wrong parameter names, import failures, API mismatch), read wrap-api.md, evaluators.md, or testing-api.md for the authoritative API reference before guessing at a fix.