AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

creating-online-evaluations

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals". Covers letting the explored data decide how many evals to create, proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge / sentiment), configuring a provider and model for an llm_judge eval (a provider key gates enabling, not creation), scoping which generations trigger it via conditions, creating disabled, verifying scope, enabling, and backfilling past data. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--creating-online-evaluations at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 2

references/evaluation-payload.md

Evaluation create payload reference

Full field reference for posthog:llma-evaluation-create. The evaluation_config and output_config schemas below are rendered from the backend Pydantic models at build time, so they can't drift.

Top-level fields

| Field | Required | Notes | | --------------------- | -------------- | ------------------------------------------------------------------------------------- | | name | yes | Up to 400 chars. | | description | no | Defaults to "". | | evaluation_type | yes | "hog", "llm_judge", or "sentiment". | | evaluation_config | yes | Shape depends on evaluation_type (below). | | output_type | yes | "boolean", "numeric", or "categorical" for hog/llm_judge; "sentiment" for sentiment. | | output_config | no | { "allows_na": bool, "true_is_failure": bool } for boolean; numeric and categorical settings below; {} for sentiment. | | model_configuration | llm_judge only | Provider + model; key ID optional. Rejected on hog/sentiment. | | target | no | "generation" (default), "trace", or "session". Sentiment supports only "generation". | | target_config | trace/session only | Settle config discriminated on strategy (below); defaults to a 30-minute fixed window for trace, and to a 1-hour inactivity window for session. | | conditions | no | Trigger condition sets (below). For traces, conditions match the triggering generation. | | enabled | no | Defaults to false. Create disabled, then flip with llma-evaluation-update. |

Valid (evaluation_type, output_type) pairs: (hog, boolean), (llm_judge, boolean), (hog, numeric), (llm_judge, numeric), (hog, categorical), (llm_judge, categorical), (sentiment, sentiment). The output type cannot change after creation.

target and target_config

  • "target": "generation" runs once for each matching generation and uses an empty target_config.

  • "target": "trace" runs once for the whole trace. target_config picks the settle strategy:

    • { "strategy": "fixed_window", "window_seconds": 10..7200 } waits a fixed delay after the first matching generation (defaults to 1800). A config without a strategy key means this.
    • { "strategy": "inactivity", "quiet_period_seconds": 10..1800, "max_age_seconds": 60..7200 } evaluates once no new activity arrived for the quiet period (defaults to 300), capped at max_age_seconds (defaults to 7200, must be at least the quiet period) from the first one.
  • "target": "session" runs once for the whole $ai_session_id session, after the session settles. target_config picks the settle strategy with session-sized bounds:

    • { "strategy": "inactivity", "quiet_period_seconds": 10..86400, "max_age_seconds": 60..604800 } evaluates once no new session activity arrived for the quiet period (defaults to 3600), capped at max_age_seconds (defaults to 86400, must be at least the quiet period) from the first one. This is the default strategy for a session target.
    • { "strategy": "fixed_window", "window_seconds": 10..604800 } waits a fixed delay after the first matching generation (defaults to 1800).
    • A session evaluation only fires for events that carry $ai_session_id. Producers set it on every generation or on none, so an SDK that never sets it will never trigger a session evaluation. $ai_session_id is not $session_id, PostHog's product-analytics session; the two are unrelated.
    • A session is evaluated at most once per evaluation, for as long as the completed run stays inside Temporal's retention window. A session that resumes long after being evaluated may be evaluated again.
  • Conditions always match generation properties. For a trace or session target, that matching generation schedules the eventual whole-unit evaluation.

  • Sentiment evaluations cannot use the trace or session target.

evaluation_config by type

llm_judge

{
  "description": "Configuration for LLM judge evaluations",
  "properties": {
    "prompt": {
      "description": "Evaluation criteria prompt",
      "minLength": 1,
      "title": "Prompt",
      "type": "string"
    },
    "judge_method": {
      "anyOf": [
        {
          "enum": [
            "llm",
            "decision"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Judge method. Omit to keep the saved method; null restores automatic routing.",
      "title": "Judge Method"
    },
    "reasoning_level": {
      "anyOf": [
        {
          "enum": [
            "minimal",
            "low",
            "medium",
            "high"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "How much the judge model reasons before it answers. Null uses the model's default.",
      "title": "Reasoning Level"
    }
  },
  "required": [
    "prompt"
  ],
  "title": "LLMJudgeConfig",
  "type": "object"
}

hog

{
  "description": "Configuration for Hog code evaluations",
  "properties": {
    "source": {
      "description": "Hog source code",
      "minLength": 1,
      "title": "Source",
      "type": "string"
    },
    "bytecode": {
      "description": "Compiled bytecode (set automatically on save)",
      "items": {},
      "title": "Bytecode",
      "type": "array"
    }
  },
  "required": [
    "source"
  ],
  "title": "HogEvalConfig",
  "type": "object"
}

bytecode is compiled and written by the server on save — never pass it. Send only source.

sentiment

{
  "description": "Configuration for sentiment evaluations.\n\nThe classifier is an English-trained model, so labels are unreliable for other languages. A\nmultilingual agent should use an llm_judge evaluation instead. See\nposthog/temporal/ai_observability/sentiment/README.md.",
  "properties": {
    "source": {
      "const": "user_messages",
      "default": "user_messages",
      "description": "Text source used for sentiment classification.",
      "title": "Source",
      "type": "string"
    }
  },
  "title": "SentimentEvalConfig",
  "type": "object"
}

output_config

boolean output

{
  "additionalProperties": false,
  "description": "Configuration for boolean output type",
  "properties": {
    "allows_na": {
      "default": false,
      "title": "Allows Na",
      "type": "boolean"
    },
    "true_is_failure": {
      "default": false,
      "title": "True Is Failure",
      "type": "boolean"
    },
    "decision_threshold": {
      "anyOf": [
        {
          "maximum": 1,
          "minimum": 0,
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Decision Threshold"
    }
  },
  "title": "BooleanOutputConfig",
  "type": "object"
}

allows_na: true lets the evaluator return N/A in addition to true/false. N/A differs from an execution error that skips a run. true_is_failure: true treats a raw true result as a failure and a raw false result as a pass. Leave it false for the default mapping.

numeric output

{
  "$defs": {
    "NumericPassingRule": {
      "additionalProperties": false,
      "properties": {
        "operator": {
          "enum": [
            "gte",
            "lte"
          ],
          "title": "Operator",
          "type": "string"
        },
        "threshold": {
          "title": "Threshold",
          "type": "number"
        }
      },
      "required": [
        "operator",
        "threshold"
      ],
      "title": "NumericPassingRule",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "min": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Min"
    },
    "max": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Max"
    },
    "step": {
      "anyOf": [
        {
          "exclusiveMinimum": 0,
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Step"
    },
    "allows_na": {
      "default": false,
      "title": "Allows Na",
      "type": "boolean"
    },
    "passing_rule": {
      "anyOf": [
        {
          "$ref": "#/$defs/NumericPassingRule"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "title": "NumericOutputConfig",
  "type": "object"
}

Return a finite number, not a boolean or numeric string. Optional min and max are inclusive; an out-of-bounds score skips that run. step guides scoring without rounding results. allows_na defaults to false; set it to true to allow null from Hog. Set passing_rule to {"operator": "gte", "threshold": 7} (at least 7) or use lte (at most). Without a passing rule, scores remain ungraded and reports are unavailable. Changing the rule reinterprets historical scores; previously generated reports stay unchanged.

categorical output

{
  "$defs": {
    "CategoricalOption": {
      "additionalProperties": false,
      "properties": {
        "key": {
          "maxLength": 128,
          "minLength": 1,
          "pattern": "^[a-z0-9]+(?:[_-][a-z0-9]+)*$",
          "title": "Key",
          "type": "string"
        },
        "label": {
          "maxLength": 256,
          "minLength": 1,
          "title": "Label",
          "type": "string"
        }
      },
      "required": [
        "key",
        "label"
      ],
      "title": "CategoricalOption",
      "type": "object"
    },
    "CategoricalPassingRule": {
      "additionalProperties": false,
      "properties": {
        "categories": {
          "items": {
            "type": "string"
          },
          "title": "Categories",
          "type": "array"
        }
      },
      "required": [
        "categories"
      ],
      "title": "CategoricalPassingRule",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "options": {
      "items": {
        "$ref": "#/$defs/CategoricalOption"
      },
      "maxItems": 100,
      "minItems": 1,
      "title": "Options",
      "type": "array"
    },
    "selection_mode": {
      "default": "single",
      "enum": [
        "single",
        "multiple"
      ],
      "title": "Selection Mode",
      "type": "string"
    },
    "allows_na": {
      "default": false,
      "title": "Allows Na",
      "type": "boolean"
    },
    "passing_rule": {
      "anyOf": [
        {
          "$ref": "#/$defs/CategoricalPassingRule"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "required": [
    "options"
  ],
  "title": "CategoricalOutputConfig",
  "type": "object"
}

options holds 1 to 100 {key, label} pairs. Keys are unique lowercase identifiers (a-z, 0-9, _, -) and are what results store; labels are for display. Keep a key stable when you rename its label. selection_mode is single (each result has exactly one key) or multiple (zero or more keys). Hog returns a list of keys, or a single key string in single mode; null is N/A when allows_na is true. Set passing_rule to {"categories": ["no_problem"]}: a result passes when it has at least one key and every key is a passing key, so an empty result fails. In multiple mode, {"categories": []} passes only empty results, which fits detectors where every key is a problem. Without a passing rule, results are recorded but there are no pass rates or reports. Changing the rule reclassifies historical results; previously generated reports stay unchanged.

sentiment output

Empty object: {}.

model_configuration (llm_judge only)

| Field | Required | Notes | | ----------------- | -------- | ------------------------------------------------------------------------------ | | provider | yes | One of openai, anthropic, gemini, openrouter, fireworks, azure_openai, together_ai. | | model | yes | Model id, e.g. gpt-5-mini. Validate against llma-evaluation-judge-models. | | provider_key_id | no | UUID of an ok-state key for the same provider, or null to use the active team key. |

provider and model are required. Pin provider_key_id to run on one specific key, or set it to null to run on the team's active key — both save a valid draft. To enable an unpinned eval, the active key must be ok and use the same provider (llma-evaluation-config-get shows it). A disabled draft may be saved without a usable key, but it cannot be tested or enabled until a key can be resolved.

conditions

A list of condition sets. OR between sets, AND within a set's properties. Omitting conditions (or an empty list) matches every $ai_generation. A generation target evaluates each match; a trace target evaluates each matching trace once.

| Field | Required | Notes | | -------------------- | -------- | --------------------------------------------------------------------- | | id | yes | Stable string identifier for the set (e.g. "default"). | | rollout_percentage | no | 0–100, defaults to 100. The sampling rate the dispatcher reads. | | properties | no | Flat list of PostHog property filters, AND-ed together. |

Each property filter: { "key": "...", "type": "event" | "person", "operator": "...", "value": ... }. Common operators: exact, is_not, icontains, not_icontains, regex, gt, lt, is_set, is_not_set. There is no time/date field inside conditions — scope by event timestamp upstream if needed, and sample volume with rollout_percentage.

"conditions": [
  {
    "id": "gpt-only",
    "rollout_percentage": 50,
    "properties": [
      { "key": "$ai_model", "type": "event", "operator": "icontains", "value": "gpt" },
      { "key": "$ai_is_error", "type": "event", "operator": "exact", "value": ["false"] }
    ]
  }
]

Full examples

Hog (no provider key required)

{
  "name": "Reply is under 2,000 characters",
  "evaluation_type": "hog",
  "evaluation_config": { "source": "for (let i, item in evaluation_events) { if (item.event == '$ai_generation' and length(item.output_text) >= 2000) { return false } } return true" },
  "output_type": "boolean",
  "output_config": { "allows_na": false },
  "target": "generation",
  "target_config": {},
  "conditions": [{ "id": "default", "rollout_percentage": 100, "properties": [] }],
  "enabled": false
}

The same Hog source works for a whole trace. Change only the target fields:

{
  "target": "trace",
  "target_config": { "strategy": "fixed_window", "window_seconds": 1800 }
}

Or evaluate when the trace goes quiet instead of after a fixed delay:

{
  "target": "trace",
  "target_config": { "strategy": "inactivity", "quiet_period_seconds": 300, "max_age_seconds": 7200 }
}

The same Hog source also works for a whole session, once no new activity has arrived for the quiet period:

{
  "target": "session",
  "target_config": { "strategy": "inactivity", "quiet_period_seconds": 3600, "max_age_seconds": 86400 }
}

New Hog source should use the shared evaluation_events and target globals. Top-level generation globals such as input, output, properties, and event, plus the trace-only events and trace globals, remain available for compatibility with saved evaluators. Session evaluations do not carry the legacy events / trace globals: session Hog source only receives evaluation_events and target. item.input_text and item.output_text are best-effort readable projections of common provider payloads; use raw item.input and item.output when exact structure matters.

LLM judge

{
  "name": "Response stays on-topic",
  "description": "Fails if the assistant changes topic from the user's question",
  "evaluation_type": "llm_judge",
  "evaluation_config": { "prompt": "Return true if the assistant's reply stays on the user's topic, false if it changes subject. Return N/A if the user did not ask a question." },
  "output_type": "boolean",
  "output_config": { "allows_na": true },
  "model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": null },
  "target": "generation",
  "target_config": {},
  "conditions": [{ "id": "default", "rollout_percentage": 100, "properties": [] }],
  "enabled": false
}

provider_key_id: null runs on the team's active key and always saves a valid draft. Pin it to an ok-state key uuid from llma-provider-key-list to run on one specific key. To enable an unpinned eval, the team's active key must be ok and its provider openai.