references/ai_product_evals.md
AI Product Evals — the new PRD quality contract
The single most-demanded new PM competency of 2025–2026, and the biggest coverage gap this domain had. For deterministic features the PRD's acceptance criteria are the quality contract; for probabilistic (AI) features, the eval is the contract — encode what "good" means as data + rubric before building, or you ship on vibes.
Evals are the new PRD
The PM owns three artifacts per AI feature:
- Golden set — a labeled collection of real inputs with expected-quality outputs, covering every intent the feature claims to handle plus the known failure modes (hallucination, refusal, off-topic, unsafe). Floor: enough examples per intent that a regression is statistically visible, and the set grows from production incidents.
- Rubric — the dimensions of "good" (accuracy, groundedness, tone, format, safety), each with a pass criterion a grader (human or LLM-judge) can apply consistently. Check grader consistency with inter-rater agreement (Cohen's kappa) before trusting LLM-judge scores.
- Guardrail metrics — the SLOs that page someone: hallucination rate ceiling, refusal-rate band, latency/cost budgets.
Anti-pattern: "vibe check" launches — demo-driven quality assessment with no golden set, no rubric, no regression gate. It is the AI equivalent of shipping without tests.
Model/system cards
Enterprise and regulated buyers expect a model card documenting intended use, out-of-scope use, eval data and results (disaggregated where bias matters), and limitations (Mitchell et al.'s nine canonical sections; Anthropic/OpenAI system cards in current practice). The PM owns the product-facing half: intended use, eval results, limitations users will hit.
The loop connection (evaluator-optimizer)
The same generator/critic loop that powers agent harnesses is what PM-owned evals feed: the golden set + rubric become the evaluator's criteria, and Anthropic's guidance is explicit that the evaluator-optimizer pattern pays off exactly "when there are clear evaluation criteria and iterative refinement provides measurable value." Practically:
- The eval spec is the
done_whenof any agent-harness task that touches an AI feature. - Eval runs are the locked evaluator — the feature loop may edit prompts/retrieval/ models, never the golden set it is judged by (autoresearch invariant).
- Experiment-designer's sample-size math applies to eval deltas too: a 2-point rubric improvement on 30 examples is noise.
Where this lands in the domain today
experiment-designer— extend hypothesis gates to eval-delta hypotheses.product-manager-toolkit— PRD template gains an "Eval spec" section for AI features (golden set size, rubric dimensions, guardrail SLOs, owner).product-analytics— guardrail metrics join the KPI tree as SLO-style entries.
Sources
- Lenny's Newsletter, "Beyond vibe checks: A PM's complete guide to evals" — https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete
- Braintrust, "Evals for PMs" — https://www.braintrust.dev/blog/evals-for-pms
- Aakash Gupta, "AI Evals" — https://www.news.aakashg.com/p/ai-evals
- Mitchell et al., "Model Cards for Model Reporting" (FAT* 2019) — https://arxiv.org/abs/1810.03993
- Anthropic, "Building Effective Agents" (evaluator-optimizer applicability) — https://www.anthropic.com/research/building-effective-agents
- Jacob Cohen, "A Coefficient of Agreement for Nominal Scales" (1960) — kappa as the inter-rater agreement statistic
- This repo:
engineering/autoresearch-agent(locked evaluator),engineering/self-eval(anti-inflation scoring)