EVAL.md
PR Writer Evaluation
Use these AXIS evals to check whether pr-writer produces concise reviewer-facing PR titles and bodies.
Quick Run
Run the working set:
npx @netlify/axis run --config skills/pr-writer/evals/axis.config.json
Run one behavior case:
npx @netlify/axis run \
--config skills/pr-writer/evals/axis.config.json \
--scenario concise-docs-pr
Inspect results:
npx @netlify/axis reports latest --config skills/pr-writer/evals/axis.config.json
npx @netlify/axis reports latest --config skills/pr-writer/evals/axis.config.json --html
Compare to a stored baseline:
npx @netlify/axis run \
--config skills/pr-writer/evals/axis.config.json \
--compare-baseline
What This Tests
| Layer | Evidence | |-------|----------| | structure | AXIS transcript, final message, and judge checks | | deterministic checks | title format, no default template headings, no validation dump | | model review | reviewer usefulness and PR-body concision |
Case Set
| Case | Purpose |
|------|---------|
| concise-docs-pr | docs-only skill changes should stay concise and omit template or validation boilerplate |
Add cases for bug fixes, breaking contracts, and broad generated diffs when changing those parts of SKILL.md.
Adoption Gate
Adopt a pr-writer change only when:
- AXIS runs complete.
- judge checks have no critical failure.
- output title uses an allowed conventional type.
- output body avoids default
Summary/Test Plantemplates. - validation is omitted unless it changes reviewer risk.