AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

setup

Use for /autoresearch-agent:setup, or to start a new autoresearch experiment: target, eval command, metric, direction, evaluator.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--setup at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 1

SKILL.md

/autoresearch-agent:setup — Create New Experiment

Set up a new autoresearch experiment with all required configuration.

Usage

/autoresearch-agent:setup                                    # Interactive mode
/autoresearch-agent:setup engineering api-speed src/api.py "pytest bench.py" p50_ms lower
/autoresearch-agent:setup --list                             # Show existing experiments
/autoresearch-agent:setup --list-evaluators                  # Show available evaluators

What It Does

If arguments provided

Pass them directly to the setup script:

python {skill_path}/scripts/setup_experiment.py \
  --domain {domain} --name {name} \
  --target {target} --eval "{eval_cmd}" \
  --metric {metric} --direction {direction} \
  [--evaluator {evaluator}] [--scope {scope}]

If no arguments (interactive mode)

Collect each parameter one at a time:

  1. Domain — Ask: "What domain? (engineering, marketing, content, prompts, custom)"
  2. Name — Ask: "Experiment name? (e.g., api-speed, blog-titles)"
  3. Target file — Ask: "Which file to optimize?" Verify it exists.
  4. Eval command — Ask: "How to measure it? (e.g., pytest bench.py, python evaluate.py)"
  5. Metric — Ask: "What metric does the eval output? (e.g., p50_ms, ctr_score)"
  6. Direction — Ask: "Is lower or higher better?"
  7. Evaluator (optional) — Show built-in evaluators. Ask: "Use a built-in evaluator, or your own?"
  8. Scope — Ask: "Store in project (.autoresearch/) or user (~/.autoresearch/)?"

Then run setup_experiment.py with the collected parameters.

Listing

# Show existing experiments
python {skill_path}/scripts/setup_experiment.py --list

# Show available evaluators
python {skill_path}/scripts/setup_experiment.py --list-evaluators

Built-in Evaluators

| Name | Metric | Use Case | |------|--------|----------| | benchmark_speed | p50_ms (lower) | Function/API execution time | | benchmark_size | size_bytes (lower) | File, bundle, Docker image size | | test_pass_rate | pass_rate (higher) | Test suite pass percentage | | build_speed | build_seconds (lower) | Build/compile/Docker build time | | memory_usage | peak_mb (lower) | Peak memory during execution | | llm_judge_content | ctr_score (higher) | Headlines, titles, descriptions | | llm_judge_prompt | quality_score (higher) | System prompts, agent instructions | | llm_judge_copy | engagement_score (higher) | Social posts, ad copy, emails |

After Setup

Report to the user:

  • Experiment path and branch name
  • Whether the eval command worked and the baseline metric
  • Suggest: "Run /ar-run {domain}/{name} to start iterating, or /ar-loop {domain}/{name} for autonomous mode."