AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

working-with-autoresearch

Use PostHog Autoresearch, agent-driven predictive models: ask "who will do X in the next N days?", let an agent train a model, then score people daily into a person property and `autoresearch_prediction` events. Use when a user wants to predict churn, activity, return, feature adoption, conversion or any event or action per person; create or validate a model from a template; start training or score now; read model quality (holdout vs realized AUC, lift, calibration, likelihood segments, coverage); steer training with suggestions; act on predictions with cohorts, feature flags, workflows or SQL; or diagnose a model that fails to train, scores everyone the same, or stops scoring. Trigger on "autoresearch", "predictive model", "propensity score", "churn prediction", "likely to convert", "who will do X", "predicted_p_", "autoresearch_prediction", "realized AUC", "champion model".

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--working-with-autoresearch at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 5

references/troubleshooting-and-steering.md

Troubleshooting and steering

First look

For any "something is wrong with my model" question, read these three, in this order:

  1. posthog:autoresearch-retrieve: status, champion, last scored time, coverage.
  2. posthog:autoresearch-training-runs-list: the newest training runs, their status, iteration_count, best_holdout_score and error.
  3. posthog:autoresearch-runs-list: the newest scoring and validation runs, their status, rows_scored and error.

Symptoms

| Symptom | Likely cause | What to do | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Training run shows 0 iterations but is running | Normal. Counters are written only when the run completes. | Wait. Check again later. | | Training run failed with 0 iterations | Usually the environment, not the question: access, quota or a sandbox problem. | Read error. Confirm the user has PostHog Desktop access. Retry once from the model page. | | posthog:autoresearch-train-create returns 403 from a sandbox | The call came from inside a sandbox, such as a PostHog AI cloud task. | Ask the user to start training from the model's page. Do not retry. | | posthog:autoresearch-train-create returns 403 about Desktop access | The user has no PostHog Desktop access, which cloud training needs. | The model page refuses too. The user needs Desktop access first. | | posthog:autoresearch-train-create returns 400 | A run is already live, the model is paused, its creator or target is no longer valid, or an action target and a key without action:read. | Read the message. Wait for the live run, resume in the UI, add the action:read scope, or recreate the model only when the target or creator is gone. | | Training completed but no champion | No experiment produced a usable model, or none beat the current champion. | Read the run's summary with posthog:autoresearch-training-runs-history. Check the base rate and positive count. | | Scores barely vary at all | An identifier mismatch between the features and the people being scored. | Check the spread with the quantile query in using-predictions.md first: on a rare target, healthy scores can all be small. If the quantiles are the same value, read the champion's feature SQL with posthog:autoresearch-models-retrieve. Every row must be keyed on the person id. | | Holdout AUC above about 0.97 | The features likely see the outcome, or a proxy that happens because of it. | Read the feature importances. If the top feature is the target or a step after it, suggest removing it. | | Scoring run failed | The feature query was too heavy, or a transient error. | Read error. After two scheduled days in a row of repeatable failures, the product starts a small rescue training run to find a champion that can score. It can fail too, so check the next training runs rather than waiting. A repeated memory error means the features scan too much: steer the next run toward shorter windows. | | Fewer people in a cohort than rows_scored | Some scored people could not be resolved to a current distinct ID. They get an event but no property. | Expected for a small share. Use the events for analysis that must include everyone. | | Population only partly scored | Rolling batches on a large population. | Expected. Read coverage on posthog:autoresearch-retrieve. | | No realized AUC before the horizon | The horizon has not passed since the champion's first scoring. | Wait. first_check_expected_at on posthog:autoresearch-retrieve says when. | | No realized AUC after the horizon | Each validated date had only one outcome class (warning single_class_no_auc). | Explain that AUC needs dates with both outcomes. More dates fill it in. On a very rare target, consider a longer horizon. | | Realized AUC drops over several dates | Behavior shifted, or the training population no longer matches who is scored. | Compare training and inference populations. Retrain, or steer with a suggestion. |

Steering with suggestions

A suggestion is a note to the training agent. It reads queued suggestions at the start of the next training run.

  • posthog:autoresearch-suggestions-create with a free-text prompt. Good suggestions name a hypothesis: "account for day-of-week, the target spikes on Mondays", "try features from the billing page events", "drop the recency feature, I suspect it leaks".
  • priority: "try_next" makes the agent act on it before its own ideas. The default consider is advisory.
  • After the next run, posthog:autoresearch-suggestions-list shows how the agent handled each one: acted_on, picked_up or dismissed, with a one-line agent_response.
  • Suggestions do not start a run. Start one from the model page, or with posthog:autoresearch-train-create after the user confirms.

The agent decides whether a suggestion helps, and the server still picks the champion by score. A suggestion that makes the model worse is recorded and dropped.

When to start over

Create a new model instead of retraining when:

  • The target, horizon or population should change. These are frozen once a model exists.
  • The validation counts were marginal and the realized AUC has stayed near 0.5 for many dates.
  • The question no longer matches what the team acts on.

Leave the old model in place until the new one has realized results, then pause or delete the old one in the UI.