AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

exploring-mcp-tool-original-user-motive

Build a starting-point taxonomy for an MCP tool — what users were trying to accomplish before they reached the tool — and publish it as a PostHog notebook. Reconstructs each session's goal from its opening tool calls, then clusters those goals into named categories with size, share, and facet mix. Use when the user asks "why do people use this tool?", "what are users actually trying to do?", "what problem brings people here?", "where do these sessions start?", "segment usage of <tool> by goal", or wants a Clio-style taxonomy of MCP usage. Complements exploring-mcp-intent-clusters, which groups what agents did per call rather than why the session began. The agent running this skill writes the goal labels itself, reading the corpus query output session by session — the bundled scripts cover the mechanical facets but measurably lose the goal's altitude, so do not delegate that field to them.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read posthog--exploring-mcp-tool-original-user-motive at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

SKILL.md · 24000

nl


This embeds each distinct intention with `text-embedding-3-small` and prints the closest pairs. It merges nothing — semantic closeness is not a merge instruction, and you have to read each pair and decide whether it is one job or two.

On the `workflows-create` run, 105 intentions produced three pairs above 0.80:

| Similarity | Pair                                                                   | Verdict                                                                                |
| ---------: | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|      0.819 | `investigate a product error` / `investigate a production error`       | same job, merge                                                                        |
|      0.805 | `build a support automation` / `build a support reply automation`      | same job, merge                                                                        |
|      0.805 | `build a recurring digest email` / `build a recurring digest to slack` | one job, two destinations — **merge**, and let the destination facet carry the channel |

Two merges out of 105, touching 8 sessions. Small, but the point is that a run without the audit reports 105 intentions when the honest count is 103, and nothing in the pipeline would have said so.

**Watch the ceiling, not just the flag count.** Highest similarity here was 0.819, so a threshold of 0.86 flags nothing and reads exactly like a clean result. The script prints the closest pairs regardless, so an empty flag list is never mistaken for verified.

This is also the only embedding step in the skill, and it is deliberately not doing the grouping — see "Why the grouping is by hand" in [`references/notebook-assembly.md`](references/notebook-assembly.md).

### 5. Publish the notebook

Full cell-by-cell recipe, including the facets cell and the sandbox gotchas: [`references/notebook-assembly.md`](references/notebook-assembly.md).

Shape:

1. `notebooks-create-markdown` — title and a short method paragraph
2. `notebooks-configure-compute` — 4 cores / 8 GB, **before** the first Python cell
3. `notebooks-add-cell` (sql) — the corpus, one row per session with its caller and org
4. `notebooks-add-cell` (sql) — caller share for the tool, over `$mcp_tool_call`, so the automated traffic you filtered out stays visible
5. `notebooks-add-cell` (python) — the facets inlined once, keyed on `sid`, as `(sid, starting_intention, theme, data_touched, <third facet>)`
6. `notebooks-add-cell` (sql) — starting intentions, a `GROUP BY` over that frame
7. `notebooks-add-cell` (sql) — themes, the same frame one level up, listing the intentions each theme holds
8. `notebooks-add-cell` (python) — caller and org per session, a second literal keyed on the same `sid`
9. `notebooks-add-cell` (sql) — intentions per org, joining those two frames, carrying the theme
10. `notebooks-add-cell` (sql) — the concentration checks from step 6
11. `notebooks-add-cell` (markdown, optional) — example sessions resolved to trace URLs, per "Linking an intention to real sessions"
12. `notebooks-add-cell` (markdown) — findings, and the skew correction from step 3

**Cells 6 and 7 carry no caller and no org.** The taxonomy states what people came to do. Mixing a population column into it answers two questions in one table and answers both worse. Cell 8 is where the two dimensions meet, and it is the only place they should.

**Key both Python frames on `sid`. Do not pre-aggregate them.** One row per distinct facet combination is smaller, and it is a dead end: with a few hundred orgs mostly holding one session each, adding the org to the combination key inflates it to roughly the session count anyway. Keyed on `sid`, every later cut is a join between the two frames and the transcription involves no counting.

**The caller and org have to be a second literal, not a join to the corpus cell.** A SQL cell that another cell joins must materialize into the notebook kernel, and a query grouping 90 days of `$mcp_tool_call` by session id exceeds the materialization caps. Narrowing the window is not the fix — it changes which sessions the corpus holds and breaks the match with the labelled snapshot. See "Publish the population as a second literal" in [`references/notebook-assembly.md`](references/notebook-assembly.md).

**Assert the two frames cover the same sessions.** `assert set(population['sid']) == set(facets['sid'])` is the only check that catches a transcription slip across two hand-written literals, and it is cheap.

Pick the `dataframe_name` when you add the cell. `notebooks-update-cell` takes only `code`, so renaming a published frame later means deleting the cell and adding it again.

### 6. Check concentration before writing findings

Run this over the corpus frame, on both dimensions, and let the answer decide whether a deeper analysis exists to do:

```sql
WITH per_org AS (SELECT org, count(*) AS n FROM corpus GROUP BY org),
ranked AS (SELECT org, n, row_number() OVER (ORDER BY n DESC) AS rk FROM per_org)
SELECT
    (SELECT count(*) FROM corpus) AS sessions,
    count(*) AS orgs,
    round(100.0 * sum(case when rk <= 1 then n else 0 end) / (SELECT count(*) FROM corpus), 1) AS top1_pct,
    round(100.0 * sum(case when rk <= 10 then n else 0 end) / (SELECT count(*) FROM corpus), 1) AS top10_pct,
    sum(case when n = 1 then 1 else 0 end) AS orgs_with_one_session
FROM ranked

The check has to be allowed to say no. On a 90-day workflows-create corpus the top org held 5.6% of sessions, the top ten held 21.2%, and roughly three fifths of orgs appeared exactly once. That is a long tail. No dominant-customer story exists, the split below does not run, and the honest move is to report the shape and continue.

When one caller or a few orgs do dominate, split the taxonomy rather than describing it. A large share is not a finding by itself. What changes a product decision is whether that population wants different things from everyone else. Recompute the intentions table for the segment and for the remainder, put them side by side, and read the differences.

Rough triggers, as a starting point rather than a rule:

  • one caller above 40% of organic sessions
  • the top ten orgs above half of them
  • any single org above 10%

One case deserves the split even when every share looks unremarkable: a theme whose org list is one org repeating. That is one customer's workflow rather than a pattern, and the session count hides it completely.

7. Read the result honestly

  • The starting intentions are the finding. Themes are a hand-assigned rollup over them; say so, and keep the intentions table available for a reader who disagrees with a grouping.
  • Read the per-org table for whether a theme is a pattern or one customer. Twenty sessions from twenty orgs and twenty from one org are different findings, and the theme table alone cannot tell them apart.
  • Read the caller table for whether one population exists or several. A theme dominated by one surface is a different product than a theme spread across all of them.
  • An intention or theme under 5 sessions is not a finding. The code suppresses them and reports the count.
  • Report shares against the corpus you actually labelled, not the raw session count.
  • Distinct-intention count against session count is a useful shape signal on its own: near 1:1 means every session is a one-off, which is a different product problem from a few intentions repeating.
  • If the largest intentions are all one automated program, the honest headline is "this tool is mostly automation", not a user taxonomy.

Linking an intention to real sessions

Every corpus row carries $mcp_session_id, and that reaches AI observability. The join runs from the trace side, not the tool-call side — $mcp_tool_call carries no $ai_trace_id at all, but $ai_generation carries $mcp_session_id.

SELECT
    toString(properties.$mcp_session_id) AS sid,
    toString(properties.$ai_trace_id) AS trace_id,
    count() AS generations
FROM events
WHERE event = '$ai_generation'
  AND timestamp > now() - INTERVAL 30 DAY
  AND coalesce(toString(properties.$mcp_session_id), '') != ''
  AND toString(properties.$mcp_session_id) IN (SELECT toString(sid) FROM <your corpus>)
GROUP BY sid, trace_id

Build the link as https://us.posthog.com/project/<project_id>/ai-observability/traces/<trace_id>. The query params the UI adds when you click through are optional.

Coverage tracks the population worth studying. On the notebooks-create corpus, roughly half of organic external sessions resolved to a trace, about three quarters of organic staff sessions, and effectively none of the wizard sessions. Automated traffic is invisible here; the sessions you actually want are the ones that resolve.

What the trace does and does not contain

Every generation attached to those sessions was $ai_product = 'mcp', $ai_span_name = 'execute-sql'. The trace shows PostHog's own server-side query work during the session — what the agent asked the warehouse and what came back.

That is genuine evidence for an intention: a concrete example of what the work looked like. It is not the user's conversation.

There is still no user message. $mcp_intent is prose the agent wrote about its own call, and the person's actual words stay in the harness — Claude Code, Cursor, the chat client — which never sends them to PostHog. If someone asks for "the user's message", a trace link is the nearest honest substitute, and worth naming as a substitute rather than passing off as the thing asked for.

/mcp-analytics/sessions takes no session id, so there is no per-session deep link on the MCP side either. The trace link is the only clickable route into one session.

Two query traps that silently invert the answer

Both produced confidently wrong numbers on this corpus before being caught.

  • properties.$mcp_session_id IN (...) lets nulls through. Rows with no session id pass the filter, so a composition query returns the project's entire $ai_generation volume while reporting uniqExact(...) = 0 sessions. Always guard with coalesce(toString(properties.$mcp_session_id), '') != '' first.
  • LEFT JOIN plus countIf(joined_col != '') counts every row as matched. An unmatched join yields '', not null, and the comparison behaves unexpectedly — it reported every session as traced where the true figure was about half of them. Use set membership (countIf(sid IN (SELECT ...))) instead of a join for coverage counts.

Sanity-check any coverage number by computing it a second way before publishing it.

Privacy

Adapted from Clio's layered defense, and each layer is load-bearing:

  1. Proper nouns are stripped at extraction, not later.
  2. Clusters below 5 sessions are suppressed.
  3. Cluster labels are synthesized from member goals, never copied from one member's intent.
  4. Raw $mcp_intent strings never enter the notebook — only goals and aggregates.
  5. Customer text reaches a third-party model only for organizations that approved AI data processing.

Layers 3 and 4 are the ones people skip. Verbatim intents carry customer names, project ids, and run ids straight into a shareable document.

Layer 5: consent before any third-party model call

scripts/extract_facets.py sends the opening column — customer-authored intent text — to the OpenAI API. PostHog's own backend gates the same class of text on the writing organization's consent: intent_generation.py returns early when team.organization.is_ai_data_processing_approved is false, and failure_classification.py does the same. A local analyst script does not get a weaker rule than the product path.

So restrict the corpus to consenting organizations before running that script. The corpus query already selects org, so the filter is a list intersection once you have the approved ids. Consent lives on the Django Organization model and is not exposed in HogQL — all_posthog_organization in the warehouse carries no such column — so the ids have to come from the Django side rather than from the corpus query itself.

Two things narrow this in practice:

  • Only extract_facets.py transmits customer text. canonicalize_intentions.py and audit_intentions.py see the goal labels, which are already generalized and stripped of proper nouns by the time they exist.
  • The default path does not call the script at all. Reading the corpus yourself is the skill's recommendation on quality grounds, and it happens to keep customer text inside PostHog's own tooling. Delegating to the script is the case that adds a third-party processor.

If you cannot establish consent for the organizations in a corpus, label the goals by hand. That is the recommended path anyway.

Related skills