AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

analyzing-offline-evaluations

Inspect and compare offline AI evaluation experiments, diagnose case-level regressions, and follow scorer history across application, model, or prompt changes. Use when a user asks whether an offline run improved, why a case failed, how scores changed between runs, or how to publish externally computed offline results. Covers immutable scorer versions, comparable dataset cohorts, incomplete coverage, and selective payload reads.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--analyzing-offline-evaluations at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

references/comparing-runs.md

Comparing runs without changing the denominator

Suppose two completed runs use the same suite, dataset revision, and scorer-version UUID. Only the application revision changed. The server reports:

| Run | Passing | Successful | Error | Missing among observed items | | --------- | ------- | ---------- | ----- | ---------------------------- | | Baseline | 72 | 90 | 5 | 5 | | Candidate | 76 | 80 | 10 | 10 |

The reported pass rates are 80% and 95%, a 15 percentage-point increase among successful results. Successful coverage fell from 90 to 80 items, while errors and missing scores grew. Report both facts; the rate alone does not establish a better application. Inspect overlapping case identities and the newly unscored/error cases.

If the candidate instead uses a new scorer version with a looser threshold, the rates describe different grading rules. Read both pinned configs, explain that difference, and recommend comparing both runs under one version before attributing the change to the application. Do not average across versions or substitute the scorer's current config for the version attached to a result.

For repeated trials, the summary weights each item once. A case with ten trials has ten times the weight of a case with one trial. Report differing trial coverage; case-weighted conclusions need a separate, explicitly described calculation over complete matching data.