AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

product-skills

Use when a product request needs routing across the product-team skills (prioritization, OKRs, UX research, experiments, discovery, roadmaps), or to run the continuous discovery loop. Routes with product_goal_router.py and gates with discovery_cadence_tracker.py and ost_linter.py.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--product-skills at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 4

references/ai_product_evals.md

AI Product Evals — the new PRD quality contract

The single most-demanded new PM competency of 2025–2026, and the biggest coverage gap this domain had. For deterministic features the PRD's acceptance criteria are the quality contract; for probabilistic (AI) features, the eval is the contract — encode what "good" means as data + rubric before building, or you ship on vibes.

Evals are the new PRD

The PM owns three artifacts per AI feature:

  1. Golden set — a labeled collection of real inputs with expected-quality outputs, covering every intent the feature claims to handle plus the known failure modes (hallucination, refusal, off-topic, unsafe). Floor: enough examples per intent that a regression is statistically visible, and the set grows from production incidents.
  2. Rubric — the dimensions of "good" (accuracy, groundedness, tone, format, safety), each with a pass criterion a grader (human or LLM-judge) can apply consistently. Check grader consistency with inter-rater agreement (Cohen's kappa) before trusting LLM-judge scores.
  3. Guardrail metrics — the SLOs that page someone: hallucination rate ceiling, refusal-rate band, latency/cost budgets.

Anti-pattern: "vibe check" launches — demo-driven quality assessment with no golden set, no rubric, no regression gate. It is the AI equivalent of shipping without tests.

Model/system cards

Enterprise and regulated buyers expect a model card documenting intended use, out-of-scope use, eval data and results (disaggregated where bias matters), and limitations (Mitchell et al.'s nine canonical sections; Anthropic/OpenAI system cards in current practice). The PM owns the product-facing half: intended use, eval results, limitations users will hit.

The loop connection (evaluator-optimizer)

The same generator/critic loop that powers agent harnesses is what PM-owned evals feed: the golden set + rubric become the evaluator's criteria, and Anthropic's guidance is explicit that the evaluator-optimizer pattern pays off exactly "when there are clear evaluation criteria and iterative refinement provides measurable value." Practically:

  • The eval spec is the done_when of any agent-harness task that touches an AI feature.
  • Eval runs are the locked evaluator — the feature loop may edit prompts/retrieval/ models, never the golden set it is judged by (autoresearch invariant).
  • Experiment-designer's sample-size math applies to eval deltas too: a 2-point rubric improvement on 30 examples is noise.

Where this lands in the domain today

  • experiment-designer — extend hypothesis gates to eval-delta hypotheses.
  • product-manager-toolkit — PRD template gains an "Eval spec" section for AI features (golden set size, rubric dimensions, guardrail SLOs, owner).
  • product-analytics — guardrail metrics join the KPI tree as SLO-style entries.

Sources

  1. Lenny's Newsletter, "Beyond vibe checks: A PM's complete guide to evals" — https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete
  2. Braintrust, "Evals for PMs" — https://www.braintrust.dev/blog/evals-for-pms
  3. Aakash Gupta, "AI Evals" — https://www.news.aakashg.com/p/ai-evals
  4. Mitchell et al., "Model Cards for Model Reporting" (FAT* 2019) — https://arxiv.org/abs/1810.03993
  5. Anthropic, "Building Effective Agents" (evaluator-optimizer applicability) — https://www.anthropic.com/research/building-effective-agents
  6. Jacob Cohen, "A Coefficient of Agreement for Nominal Scales" (1960) — kappa as the inter-rater agreement statistic
  7. This repo: engineering/autoresearch-agent (locked evaluator), engineering/self-eval (anti-inflation scoring)