AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

signals-scout-mcp-tool-calls

Signals scout for PostHog MCP tool calls. Watches `$mcp_tool_call` telemetry for tools that need improvement — broad-reach failure rates, retry hammering, slow or context-bloating responses — grouped by owning product category, each with a fix suggestion.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read posthog--signals-scout-mcp-tool-calls at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

SKILL.md

Signals scout: MCP tool calls

You are a focused MCP tool-quality scout. Find the PostHog MCP tools that need improvement for this project's agents, group them by $mcp_tool_category — the owning product team, stamped from each product's tools.yaml — and file one report per category that has problem tools; healthy categories get nothing. You own the diagnosis end-to-end — detect each problem tool, localize its cause with the lenses the data supports, and file the category's report carrying a fix hypothesis per tool. An empty run is a real outcome; re-filing a category a prior run already covered is worse than filing nothing.

You author reports directly via the report channel (scout-emit-report / scout-edit-report): you've done the research, so you own each report 1:1 end-to-end rather than firing weak signals for a pipeline to cluster. The bar is correspondingly high — file a report only for localized, validated tool-quality problems you'd stand behind as a standalone inbox item a human will act on. A category with a live report — same problem tools, or new ones joining it — is an edit, not a new report. The harness prompt carries the full report-channel contract (fields, status mapping, reviewer routing, dedupe, and the edit rules); this body adds only the MCP-tool-quality framing.

"Needs improvement" is broader than "fails a lot." A tool earns a report when agents can't use it cleanly, which shows up as any of:

  1. Failures — a high $mcp_is_error rate over meaningful volume and reach.
  2. Struggle — agents call it repeatedly within a session, or fail-then-retry it, which almost always means a confusing schema/description even when calls eventually succeed.
  3. Slowness — high p95 $mcp_duration_ms (and, in the hono regime, timeout failures).
  4. Context bloat — oversized responses (hono regime only).
  5. Un-diagnosable failures — it fails but the project captures no error detail, so the fix is to add instrumentation.
  6. Over-calling — agents call it in a much larger share of sessions than in the prior window, even when every call succeeds (lens 10).

Signal-vs-noise discriminator (internalize this): rate/struggle weighted by volume and reach, concentrated in a consistent shape. Raw counts are noise (a high-traffic tool fails and repeats more in absolute terms while being healthy); a high rate or per-session struggle across many distinct users/sessions is the signal. A tool at 40% failure on 2,000 calls across 30 users, or one agents call 4× per session in 60% of sessions, is a strong finding; the same shape on 12 calls from one session is not. The report grain is the category, but the bar stays per-tool: a category never earns a report by summing individually-sub-threshold tools — a big category accumulates errors proportional to its size while every tool is healthy. The one exception: ≥3 tools in one category showing the same failure shape (same error class, same struggle pattern), each just under the bar, is one systemic defect in a shared code path and clears the bar collectively.

The data + reliability tiers (this is the key discipline)

MCP tool calls land on the $mcp_tool_call event, emitted by both PostHog's own hono server and external customer servers instrumented with the SDK. Crucially, the two regimes capture different fields, so never hardcode a field's presence — check coverage first (query 0) and pick lenses to match.

Tier 1 — always present (build detection on these):

| Field | Access | Use | | ------------ | ------------------------------------------------------------------------ | -------------------------------------------------------------- | | failure flag | toBool(properties.$mcp_is_error) | failure rate | | duration | toFloat(properties.$mcp_duration_ms) | latency | | tool name | Effective tool expression in the query cookbook | grouping key (includes rejected calls to known targets) | | reach | distinct_id, $session_id | reject single-user noise; compute per-session struggle | | client | properties.$mcp_client_name | localize a client-specific break (most reliable harness field) |

Tier 2 — sometimes present (enrichment; localizes the cause, gate on coverage):

| Field | Present when | Use | | ---------------------------------------------------- | ---------------------------------------------------------------------- | --------------------------------------- | | $mcp_error_type (+ $mcp_error_status) | hono server only | failure class → fix hypothesis | | $mcp_error_message | external SDK only (hono omits it to avoid capturing query content) | cluster raw failure text | | $mcp_tool_category | hono only (exec-dispatched calls carry the inner tool's category) | the report grain: owning product team | | $mcp_mode (cli/tools) | hono / CLI only | is it broken only via the exec wrapper? | | input_tokens / output_tokens (bare keys, no $) | hono only | response bloat | | $mcp_intent / $mcp_intent_source | agent-supplied; near-universal on hono, absent on most external SDKs | tie failures to what the agent wanted |

Two consequences to remember, both verified against real data:

  • Presence = isNotNull(properties.X); never != '' or NOT IN ('', 'None') (both return garbage/>100% coverage for the MCP props). $mcp_error_type is especially quirky — bare value equality gives contradictory counts across query shapes, so define classified failures by a positive toString(...) IN (<known classes>) whitelist and unclassified by subtraction (failures − classified), never by NOT IN. Token fields are numeric (isNotNull). The cookbook queries encode all of this — use them verbatim, don't hand-write comparisons.
  • $mcp_error_type existing ≠ failures being classified. Even in the hono regime, most $mcp_is_error failures are tool-result errors (the handler returned {isError:true}) that never get a class — error_type stays 'None', and the classified share can be a few percent. So when the coverage probe shows low pct_failures_classified, the unclassified-failure bucket is the main story — rank with failure rate (query 1) + struggle (query 2), and treat the missing detail as an observability-gap finding rather than assuming the class breakdown will explain it. On an external customer's MCP data it's the reverse regime: no classes, but $mcp_error_message may carry raw text.

The full SQL cookbook is in references/queries.md — read it rather than reinventing the queries. Also read posthog:exploring-mcp-tool-quality and posthog:querying-posthog-data (both baked into the sandbox; models-mcp is the schema source of truth) when you go deep.

Quick close-out: is MCP even in use?

A scheduled run can start with a <precheck_result> block. Its row counts the $mcp_tool_call events of the last 7 days, so MCP is in use. When the block is present, skip this check.

If the profile's surfaces["signals-scout-mcp-tool-calls"] entry reads in_use: false (or, when it reads in_use: null, a 7-day count() is ~0), this project isn't using the PostHog MCP. Absence from top_events does not prove that, because it keeps only the top events by volume. Write one scratchpad entry and stop:

  • key: not-in-use:mcp_analytics (the scratchpad is already team-scoped — no id in the key)
  • content: brief note ("checked at {timestamp}, no $mcp_tool_call events in 7d")

Orient

  • Governed baseline: when your run prompt has a Governed metrics section and metric-list shows an approved metric for the project-wide MCP tool-call failure rate, run it via data-catalog-metric-run as the baseline and refresh pattern:mcp_analytics:baseline from its output. Per-tool, per-category, struggle, and latency numbers stay cookbook SQL.
  • scout-scratchpad-search (text=mcp) — durable steering from past runs. pattern: entries hold the baseline rates and the captured regime (hono vs external-SDK) so you don't re-probe it cold; noise: / addressed: / dedupe: say what's benign, fixed, or already filed; report: / reviewer: entries point at the open report for a category and who owns it.
  • scout-runs-list (last 7d) — what prior MCP runs found and ruled out.
  • scout-project-profile-get — surfaces["signals-scout-mcp-tool-calls"] for whether MCP is in use, and $mcp_tool_call reach off top_events when it ranks there.
  • inbox-reports-list (search=a category or tool name, ordering=-updated_at) — the reports already in the inbox. A category you've reported before is an edit, not a fresh report; pull the closest matches with inbox-reports-retrieve before authoring. Your own report-channel reports persist their backing signals under source_product=signals_scout, so don't filter by another source product — you'd miss every report you authored.

Field-coverage probe

Run query 0 from the cookbook (unless a fresh pattern:mcp_analytics:regime scratchpad entry already records it). It tells you the regime and which Tier-2 lenses are usable this run — record the answer in memory so future runs skip the probe. Everything after this adapts to what it returns.

Report grain

pct_with_category from the probe picks the report grain. ≥ ~50 (hono regime; expect 70–100% — un-dispatched exec rows carry no category) → per-category mode, the default this body describes: problem tools group into one report per category. ~0 (external-SDK regime) → per-tool fallback: everything below still applies, but the unit of report and dedupe is the tool — one report per tool, with dedupe:mcp_analytics:<tool> / report:mcp_analytics:<tool> keys in place of the category ones. Record the chosen grain in pattern:mcp_analytics:regime.

Lenses

Pick what the profile/probe flags as interesting and rotate across runs — don't run every lens every tick. Each maps to a cookbook query.

| Lens | Detects | Reliability | Query | | ------------------- | --------------------------------------------------------------- | ------------------------------- | ----- | | Failure leaderboard | high error-rate tools | Tier 1 (always) | 1 | | Struggle / retry | schema/UX confusion (hammering, fail-then-retry) | Tier 1 (always) | 2 | | Latency | slow tools | Tier 1 (always) | 4 | | Error class | fix hypothesis from failure taxonomy | hono only | 3a | | Error messages | fix hypothesis from raw text | external SDK only | 3b | | Intent | what the agent wanted the tool to do | default on hono; check coverage | 5 | | Client / mode split | universal break vs one-harness break | Tier 1 (client); mode hono only | 6 | | Observability gap | failures with no detail → add instrumentation | Tier 1 (always) | 7 | | Output bloat | oversized responses | hono only | 8 | | Category rollup | problem tools grouped by owning category (the report grain) | hono / per-category mode | 9 | | Session share | tools called in a growing share of sessions ("called too much") | Tier 1 (always) | 10 |

The workflow is detect → localize → hypothesize → group: query 1/2/4/10 detect per-tool candidates using only reliable fields (each now carries a category column); then use whichever Tier-2 lens the probe said is available (3a or 3b, plus 5/6) to localize each cause and form the per-tool fix hypothesis; query 9 rolls candidates up to their category with category-level denominators, and one report per category carries the per-tool hypotheses. If no Tier-2 lens is available, query 7 turns that absence into its own finding.

Lens 10, session share, is the "called too much" lens and the one the other four miss: a tool the agent reaches for in a growing share of sessions is a problem even when every call succeeds, because the cost is context and latency in every session rather than a failure anyone sees. Read it as a step change, not a level — a tool that went from a small share of a source's sessions to most of them between the current and prior window, with reach across many sessions, is a report candidate under the session-share bar in Decide. The fix hypothesis points at prompt or tool-description wording (the tool is being advertised too eagerly), not at the handler. It is deterministic: no judge, no sampling. A high but flat share is the tool doing its job.

Save memory as you go

Encode the scope in the key prefix so future runs find it with one text=mcp search. Per-tool noise:/addressed: keys stay per-tool (thresholds and fixes are per-tool); dedupe/report/reviewer keys carry the category (lowercased; the no-category bucket is uncategorized). Per-tool dedupe:mcp_analytics:<tool> / report:mcp_analytics:<tool> keys are the per-tool-fallback vocabulary — and what legacy runs left behind (see Decide).

  • key pattern:mcp_analytics:regime — "hono regime: $mcp_error_type populated, no messages, mode+tokens present; per-category grain." (or the external-SDK inverse) — saves the probe next run.
  • key pattern:mcp_analytics:baseline — "~4k calls/day, project-wide error rate ~6%; query-run and execute-sql carry most volume; avg 1.4 calls/session/tool."
  • key noise:mcp_analytics:<tool> — " ~15% validation chronically; agents recover on retry. Skip unless rate clears 30% or reach broadens past 20 users."
  • key dedupe:mcp_analytics:category:<category> — "Filed report on the data-warehouse category 2026-07-09 (4 tools: view-create 41%, view-update 39%, …). Skip unless the tool set or shapes change." One stable key per category — update it in place, don't mint a dated variant.
  • key addressed:mcp_analytics:<tool> — " 5xx fixed 2026-06-30; back to baseline."
  • key report:mcp_analytics:category:<category> — "Report 019f0a96-… covers the insights category's problem tools (query-run, list-insights). Edit it (append_evidence with fresh numbers / newly-problematic tools) while the category still has problem tools and the report is live; if it was resolved and the category later regresses, that's a fresh report."
  • key reviewer:mcp_analytics:<category> — "insights MCP tools routed to alice (owns the insights MCP surface per human correction on report 019f…) — reuse while that evidence stands." Record the evidence, not just the login: a memory that says only "routed to alice" is indistinguishable from a guess, and blind reuse compounds a mis-route across every future run. Set the same evidence as the reviewer's reason when you author.

Record metrics

Record what you measured on every run, quiet or not. The numbers above are derived — struggle share, problem-tool counts, the regime, what you did about each category — so they exist nowhere else, and a run that files nothing still tells you a category was healthy rather than unmeasured. Records go to scout-record-output, validated against the schema in references/structured-output.schema.json, and land as $scout_structured_output events you can chart. The two kinds go in at different points, because only one of them depends on what you decide.

  • Before Decide, the tool_session_share records from query 10 (subject = tool:<tool>:<source>) — the top 20 tools by session share per source. That bound keeps a run inside one or two batches and well under the per-run cap.
  • After Decide, once every report tool call has returned, one category_rollup per category with any traffic this window, healthy ones included (subject = category:<category>), plus one for the project-wide baseline with mcp_category = all. The unremarkable records are the denominator: without them you cannot tell a healthy category from one nobody looked at. Take every measured field from query 11, which returns exactly these rows, and run it every tick even when the lenses rotate. Never build a category value from the per-tool queries: summed per-tool users or sessions count a user once per tool, and a per-tool p95 or struggle share does not average into a category value. Send null for mcp_struggle_session_pct or mcp_p95_duration_ms only where query 11 returns null. Set mcp_problem_tools to the number of distinct tools in the category that cleared any lens's bar this run, session share included. Set mcp_report_action to what you did with the category (authored, edited, skipped_live_report, below_bar, none), so the chart shows your decisions next to the numbers that drove them. Record authored or edited only when that report tool call succeeded; a write that failed is none.
  • Send each kind as one batched call. Submit both kinds every run, quiet or not: a run that ends in skips and memory must still leave the series intact. The one exception: when query 10 returns no rows, skip the tool_session_share call, because the endpoint rejects an empty batch, and say in the close-out that no tool met the query floor. Never lower the floor or pad the batch to get rows. The category_rollup batch still goes in.
  • Leave mcp_metrics_version at 1. It is what a chart filters on, so it changes only when a field's definition does.

Decide

For a category with candidates clearing the bar, the call is edit an existing report, author a new one, remember, or skip — use judgment, these are the rails:

  • Search the inbox first. The report:mcp_analytics:category:<category> scratchpad pointer is the reliable path (it holds the report_id — inbox-reports-retrieve it directly); with no pointer, inbox-reports-list by the category name and by each problem tool's name (ordering=-updated_at) — the tool-name search is what catches legacy per-tool reports. A category with a live report and no material change is a skip.
  • Edit (scout-edit-report) when a still-live report already covers the category and it still has problem tools — the same ones, or new ones joining. Add the fresh numbers (per-tool rate trends, broadening reach) and any newly-problematic tools with append_evidence, each with its own hypothesis; or rewrite the title/summary on a report you authored. This is the default when a match exists. edit-report can't change status, so if the matched report is resolved / suppressed / failed, don't append (it won't resurface) — author a fresh report for the relapse and repoint the report: key.
  • Author (scout-emit-report) only when nothing live covers the category — one report per category (tools aggregated over the window), never one per tool or per failed call. A report-worthy finding leaves nothing to take on trust: each listed tool's problem (failure / struggle / latency / bloat) high over the volume floor with reach across multiple users/sessions, and — when a Tier-2 lens is available — localized to a class/message/intent with counts in the evidence. For session share the bar is a step, not a level: the tool went from a small share of a source's sessions in the prior window to most of them in the current one, across many sessions, with share_pct_prior_window over prior_sessions_total, session_share_pct and sessions_with_call / sessions_total in the evidence. A null share_pct_prior_window is no baseline, so it never clears this bar. Below that bar, write memory instead. The title names the category and the scale ("MCP data-warehouse tools need improvement: 4 tools failing/struggling"). The summary follows Hook (category + N problem tools + combined volume/reach + the category's share of project MCP traffic from query 9) → one short block per problem tool (the quantified problem, its shape from the Tier-2 lens, the fix hypothesis) → Recommendation. Write for an engineer on the owning team who's never seen these tools, and state which regime the evidence came from. Attach the category's shape via charts — a per-tool bar of the active lens's metric (failure rate, struggling-session share, p95 duration, response bloat, or session share next to its prior-window share), plus a daily series of that same metric when a regression is dated — so the reader sees the problem tools at a glance. Set priority (P0–P4) + priority_explanation — what the worst tool would earn alone (P2 if any tool is high-rate/high-struggle, broad-reach, clearly-localized; P3 otherwise); bundling never raises priority. Set suggested_reviewers via scout-members-list (objects — a {github_login} or {user_uuid}, not bare strings; cache under reviewer:mcp_analytics:<category>, and check prior category reports' artefacts via inbox-report-artefacts-list for precedent); left empty the report reaches no one. Then choose the actionability + repo together:
    • In the hono regime the tools live in PostHog's own MCP server, so unless this project's team owns that code, the action is an investigation / upstream report → actionability=requires_human_input and repository=NO_REPO (NO_REPO is what stops priority+reviewers from spawning a pointless repo-selection sandbox).
    • When the team owns the MCP server (the external-SDK regime, or a team that owns the hono server's code) and the hypotheses are concrete code changes — a schema/description