references/troubleshooting-and-steering.md
Troubleshooting and steering
First look
For any "something is wrong with my model" question, read these three, in this order:
posthog:autoresearch-retrieve: status, champion, last scored time, coverage.posthog:autoresearch-training-runs-list: the newest training runs, their status,iteration_count,best_holdout_scoreanderror.posthog:autoresearch-runs-list: the newest scoring and validation runs, their status,rows_scoredanderror.
Symptoms
| Symptom | Likely cause | What to do |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Training run shows 0 iterations but is running | Normal. Counters are written only when the run completes. | Wait. Check again later. |
| Training run failed with 0 iterations | Usually the environment, not the question: access, quota or a sandbox problem. | Read error. Confirm the user has PostHog Desktop access. Retry once from the model page. |
| posthog:autoresearch-train-create returns 403 from a sandbox | The call came from inside a sandbox, such as a PostHog AI cloud task. | Ask the user to start training from the model's page. Do not retry. |
| posthog:autoresearch-train-create returns 403 about Desktop access | The user has no PostHog Desktop access, which cloud training needs. | The model page refuses too. The user needs Desktop access first. |
| posthog:autoresearch-train-create returns 400 | A run is already live, the model is paused, its creator or target is no longer valid, or an action target and a key without action:read. | Read the message. Wait for the live run, resume in the UI, add the action:read scope, or recreate the model only when the target or creator is gone. |
| Training completed but no champion | No experiment produced a usable model, or none beat the current champion. | Read the run's summary with posthog:autoresearch-training-runs-history. Check the base rate and positive count. |
| Scores barely vary at all | An identifier mismatch between the features and the people being scored. | Check the spread with the quantile query in using-predictions.md first: on a rare target, healthy scores can all be small. If the quantiles are the same value, read the champion's feature SQL with posthog:autoresearch-models-retrieve. Every row must be keyed on the person id. |
| Holdout AUC above about 0.97 | The features likely see the outcome, or a proxy that happens because of it. | Read the feature importances. If the top feature is the target or a step after it, suggest removing it. |
| Scoring run failed | The feature query was too heavy, or a transient error. | Read error. After two scheduled days in a row of repeatable failures, the product starts a small rescue training run to find a champion that can score. It can fail too, so check the next training runs rather than waiting. A repeated memory error means the features scan too much: steer the next run toward shorter windows. |
| Fewer people in a cohort than rows_scored | Some scored people could not be resolved to a current distinct ID. They get an event but no property. | Expected for a small share. Use the events for analysis that must include everyone. |
| Population only partly scored | Rolling batches on a large population. | Expected. Read coverage on posthog:autoresearch-retrieve. |
| No realized AUC before the horizon | The horizon has not passed since the champion's first scoring. | Wait. first_check_expected_at on posthog:autoresearch-retrieve says when. |
| No realized AUC after the horizon | Each validated date had only one outcome class (warning single_class_no_auc). | Explain that AUC needs dates with both outcomes. More dates fill it in. On a very rare target, consider a longer horizon. |
| Realized AUC drops over several dates | Behavior shifted, or the training population no longer matches who is scored. | Compare training and inference populations. Retrain, or steer with a suggestion. |
Steering with suggestions
A suggestion is a note to the training agent. It reads queued suggestions at the start of the next training run.
posthog:autoresearch-suggestions-createwith a free-textprompt. Good suggestions name a hypothesis: "account for day-of-week, the target spikes on Mondays", "try features from the billing page events", "drop the recency feature, I suspect it leaks".priority: "try_next"makes the agent act on it before its own ideas. The defaultconsideris advisory.- After the next run,
posthog:autoresearch-suggestions-listshows how the agent handled each one:acted_on,picked_upordismissed, with a one-lineagent_response. - Suggestions do not start a run. Start one from the model page, or with
posthog:autoresearch-train-createafter the user confirms.
The agent decides whether a suggestion helps, and the server still picks the champion by score. A suggestion that makes the model worse is recorded and dropped.
When to start over
Create a new model instead of retraining when:
- The target, horizon or population should change. These are frozen once a model exists.
- The validation counts were marginal and the realized AUC has stayed near 0.5 for many dates.
- The question no longer matches what the team acts on.
Leave the old model in place until the new one has realized results, then pause or delete the old one in the UI.