AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

azure-diagnostics

Debug Azure production issues using AppLens, Azure Monitor, resource health, and safe triage, including evidence-only diagnosis without live access. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--azure-diagnostics at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 42

references/functions/durable.md

Durable Functions Troubleshooting

Diagnostics for stuck orchestrations, non-determinism, and task hub conflicts.

Stuck Orchestrations

| Symptom | Cause | Fix | |---------|-------|-----| | Orchestration stuck in Running indefinitely | An awaited durable timer/sub-orchestration/activity never completes, or completes but the orchestrator never observes it due to a non-determinism error breaking replay | Query instance status; check for OrchestrationServiceException/replay errors in App Insights traces for that instanceId | | Orchestration stuck in Pending | Task hub control-queue backlog, host-storage access failure, or the Durable extension isn't loading | Check control queue depth, AzureWebJobsStorage, extension startup logs, and scale-controller traces | | Terminated/Failed but activity output looks fine | Unhandled exception in the orchestrator itself (not an activity) | Orchestrator-level exceptions fail the whole instance; wrap sub-orchestrator/activity calls in try/catch if partial failure should be tolerated |

# Check one orchestration instance
func durable get-runtime-status --connection-string-setting AzureWebJobsStorage \
  --task-hub-name TASKHUB --id <instance-id> --show-output

Non-Determinism Violations

Orchestrator functions must produce the same durable operations in the same order on replay. Pure computation is safe; nondeterministic I/O, time, randomness, and unmanaged tasks aren't.

| Anti-pattern | Why It Breaks | Fix | |--------------|----------------|-----| | Thread.Sleep / Task.Delay | Not replay-safe: uses a non-durable wait that isn't persisted, so it isn't recomputed identically on replay | context.CreateTimer() | | DateTime.Now / DateTime.UtcNow | Returns a different value on every replay | context.CurrentUtcDateTime | | Guid.NewGuid() | Generates a new value on every replay | context.NewGuid() | | Direct HTTP/DB calls in the orchestrator body | Side effects re-execute on every replay | Move to an activity function and call it via context.CallActivityAsync | | async/await on non-Durable-Task APIs (e.g. raw Task.Run, unmanaged threads) | Not tracked in the orchestration history, so it can't be replayed consistently | Only await context.Call*Async/context.CreateTimer/context.WaitForExternalEvent inside an orchestrator |

Search App Insights exceptions for NonDeterministic when an orchestration silently stalls. The extension logs NonDeterministicWorkflowException/replay errors when history and current execution diverge.

Timeout Patterns: Cancel the Loser, Don't Just Race

A common but incomplete pattern uses Task.WhenAny to add a timeout to a sub-orchestration call, but Task.WhenAny only picks a winner. It doesn't cancel the loser. If the sub-orchestration wins, an outstanding durable timer keeps the parent instance's history growing until it fires. If the timer wins, the sub-orchestration keeps running with no caller waiting on it, which can duplicate side effects on retry.

This in-process C# example cancels the timer and assigns a known child ID:

using var timeoutCts = new CancellationTokenSource();
var deadline = context.CurrentUtcDateTime.AddMinutes(5);
var childInstanceId = $"{context.InstanceId}:child";
var subTask = context.CallSubOrchestratorAsync<Result>(
    "ChildOrchestrator", childInstanceId, input);
var timeoutTask = context.CreateTimer(deadline, timeoutCts.Token);
var winner = await Task.WhenAny(subTask, timeoutTask);

if (winner == subTask)
{
    timeoutCts.Cancel(); // stop the outstanding timer from lingering in history
    return await subTask;
}

// The child is still running. Include its ID so an external monitor can terminate it.
throw new TimeoutException($"Child orchestration {childInstanceId} timed out.");

context.CreateTimer accepts a CancellationToken; canceling it on the success path allows the parent to complete without waiting for the timer. Task.WhenAny does not cancel an activity or sub-orchestration. If the child must stop, call TerminateAsync (in-process) or TerminateInstanceAsync (isolated worker) from a client-capable function using the known child ID. Do not inject or call a Durable client from orchestrator code. Termination also does not stop an activity already executing, so activity side effects must be idempotent or cooperatively cancellable.

Task Hub Conflicts

| Symptom | Cause | Fix | |---------|-------|-----| | Orchestrations from two apps interleave/corrupt each other's history | Two function apps configured with the same task hub name against the same storage backend | Assign each app a unique task hub name | | App fails to start after deployment, hub-name-related error | Task hub name isn't alphanumeric, or exceeds 45 characters | Task hub names must be alphanumeric only (letters and numbers, starting with a letter, 3-45 chars). The name prefixes storage artifacts that forbid special characters. MyApp-TaskHub-Prod is invalid; use MyAppTaskHubProd | | Duplicate/lost orchestration instances after a shared-storage migration | Multiple apps unintentionally sharing the same storage account and default task hub name | Give each app an explicit, unique hubName in host.json rather than relying on the default |

# Check whether a task hub name override is set via app settings
az functionapp config appsettings list -n APP -g RG \
  --query "[?name=='AzureFunctionsJobHost__extensions__durableTask__hubName']" -o table

# If no override is set, the hub name comes from host.json:
#   - Inspect host.json in your source repo, or
#   - Use Kudu (Advanced Tools) / zip-deployed content to view host.json in the deployed app
{
  "extensions": {
    "durableTask": {
      "hubName": "MyAppTaskHubProd"
    }
  }
}

Tip: For the Azure Storage provider, a task hub uses multiple tables, queues, and blobs. See the storage providers docs for the full schema before assuming an artifact is missing.

Purging Old Instance History

# Compute a cutoff timestamp 7 days ago, portable across GNU (Linux) and BSD (macOS) date
if CUTOFF=$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null); then
  :
else
  CUTOFF=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ)
fi

func durable purge-history --connection-string-setting AzureWebJobsStorage \
  --task-hub-name TASKHUB --created-before "$CUTOFF" \
  --runtime-status completed failed terminated canceled

Always restrict purge operations to terminal statuses. Without --runtime-status, Core Tools also purges matching active instances.

⚠️ Growing history warning: Orchestrations with >10,000 history events can hit performance/storage limits. To check whether an instance's history is large, query the History table for that instance's partition key and count rows. --num-results 1 only confirms the table has entries, it doesn't estimate row count.

Integration Testing Durable Failure Paths

  • Stuck orchestration: start an orchestration whose sub-orchestrator never completes; assert the instance stays Running and a timeout/terminate recovery path exists.
  • Non-determinism: introduce DateTime.UtcNow into an orchestrator, force a replay, and assert the extension surfaces a non-determinism error rather than silently diverging.
  • Task hub conflict: in Azurite or isolated test storage, run two apps with the same hub name and assert the collision is detected; never run this test against shared production storage.
  • Timeout/cancellation: assert that when the timer wins a Task.WhenAny race, the losing sub-orchestration is either explicitly terminated or documented as intentionally left running.