AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

instrument-llm-analytics

Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--instrument-llm-analytics at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 40

references/basics.md

AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

AI Observability basics

Canonical URL: https://posthog.com/docs/ai-observability/basics

On this page

This page covers how your LLM calls become analytics in PostHog and defines key concepts for AI Observability.

How LLM calls become events

PostHog's AI Observability works by wrapping your existing LLM provider's SDK to capture requests and responses. Your API calls still go directly to your provider, but the wrapper extracts metadata from each call and sends it to PostHog as an event.

graph LR Call["LLM call"] --> Wrapper["Wrapped SDK"] Wrapper --> Provider["LLM provider API"] Wrapper --> Event["Event"] Event --> PostHog["PostHog"]

Events sent to PostHog are called generations. A generation represents a single LLM call. For example, when you send a prompt to Claude and get a response back, that's one generation.

Generations are represented using the event name $ai_generation. Each generation captures the model, provider, input, output, token counts, latency, and cost.

The $ai_ prefix

Every event name that starts with $ai_ is an AI Observability event. This includes the names the SDKs send, like $ai_generation, $ai_span, $ai_trace, and $ai_embedding, and any other $ai_ name you send yourself.

PostHog processes every $ai_ event as an AI Observability event rather than a Product Analytics event:

  • It doesn't create or update a person profile. $set and $set_once on the event are ignored.
  • The full event, including large properties like $ai_input and $ai_output, is kept for 30 days. After that, only a copy without those properties remains. See data retention.
  • It's billed as an AI Observability event, not as a Product Analytics event.

Don't use the $ai_ prefix for events that need person profiles or Product Analytics pricing.

Generation properties

Tokens and costs

Tokens are the units LLMs use to process text. LLM providers charge based on token usage:

  • Input tokens are tokens used in the message you send to an LLM
  • Output tokens are tokens used in the message you receive from an LLM

PostHog automatically calculates costs by matching your model and provider against pricing data. We use OpenRouter's pricing as our primary source, with fallback to manually maintained pricing for additional models.

You can also set custom pricing if you have negotiated rates or use unsupported models.

Message roles

When you send messages to an LLM, each message has a role that tells the model how to interpret it:

| Role | Purpose | Example | | --- | --- | --- | | system | Instructions that define the assistant's behavior | "You are a helpful assistant that speaks like a pirate" | | user | Messages from the end user | "What's the weather today?" | | assistant | Previous model responses, used for conversation history | "Arrr, it be sunny with a chance of scurvy!" |

PostHog captures the full message array with roles intact, so you can see exactly what context the model had when it generated a response.

Grouping related events

Most LLM applications involve multiple calls. Traces, spans, and sessions let you see how they connect:

graph TD Session["Session"] Trace["Trace"] Generation1["Generation"] Span["Span"] Generation2["Generation"] Session -.-> Trace Trace --> Generation1 Trace --> Span Span --> Generation2

Here's a breakdown of this hierarchy:

| Term | Definition | Example | | --- | --- | --- | | Session | Groups multiple traces together | A user's conversation thread | | Trace | Contains generations and spans for a single request | One chatbot message and response | | Span | Tracks an operation within a trace | A retrieval step or function call | | Generation | An LLM call, tracked as $ai_generation events | Sending a prompt to Claude | | Embedding | Converts text into vectors | Vectorizing documents for RAG |

A session here usually means a conversation

In AI Observability, a session ($ai_session_id) typically represents a single conversation – one thread of back-and-forth turns with your assistant. This is different from a standard PostHog session ($session_id), like the ones you see in Session replay, which represents one continuous visit to your app.

A single PostHog session can contain multiple AI Observability sessions: a user might start several separate conversations during one visit. See sessions for more on the distinction.

See it in action

Want to see these concepts applied in real projects? Check out our demos and example projects or browse AI Observability side projects built by the PostHog team.