AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

eval-driven-dev

Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read github--eval-driven-dev at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 17

references/runnable-examples/standalone-function.md

Runnable Example: Standalone Function (No Server)

When the app is a plain Python function or module — no web framework, no server, no infrastructure.

Approach: Import and call the function directly from run(). This is the simplest case.

# pixie_qa/run_app.py
from pydantic import BaseModel
import pixie


class AppArgs(BaseModel):
    question: str


class AppRunnable(pixie.Runnable[AppArgs]):
    """Drives a standalone function for tracing and evaluation."""

    @classmethod
    def create(cls) -> "AppRunnable":
        return cls()

    async def run(self, args: AppArgs) -> None:
        from myapp.agent import answer_question
        await answer_question(args.question)

If the function is synchronous, wrap it with asyncio.to_thread:

import asyncio

async def run(self, args: AppArgs) -> None:
    from myapp.agent import answer_question
    await asyncio.to_thread(answer_question, args.question)

If the function depends on an external service (e.g., a vector store), the wrap(purpose="input") calls you added in Step 2a handle it automatically — the registry injects test data in eval mode.

When to use setup() / teardown()

Most standalone functions don't need lifecycle methods. Use them only when the function requires a shared resource (e.g., a pre-loaded embedding model, a database connection):

class AppRunnable(pixie.Runnable[AppArgs]):
    _model: SomeModel

    @classmethod
    def create(cls) -> "AppRunnable":
        return cls()

    async def setup(self) -> None:
        from myapp.models import load_model
        self._model = load_model()

    async def run(self, args: AppArgs) -> None:
        from myapp.agent import answer_question
        await answer_question(args.question, model=self._model)