Evidence labels: OBSERVED for the single incident record described below; DESIGN GUIDANCE for the detection procedure. This article reports no measured detector accuracy or universal timeout.
View the full infographic → · View this article
A process can be alive while a model call is stalled, and a model call can be active while the workflow has made no durable progress. Keep those signals separate so legitimate long work is not killed because it is quiet.
Track four different signals
- Process liveness: is the worker still scheduled and able to report?
- Provider activity: is a request pending, streaming, rate-limited or invalidated?
- Stage progress: has the job produced a new checkpoint or artifact?
- Acceptance: did a verifier confirm the requested outcome?
A heartbeat is only the first signal. It should not renew a semantic-progress deadline. A provider response is activity, not completion. A completed stage is not acceptance until its stated check passes.
Choose deadlines by stage
Give each stage an expected range, checkback time and useful-long-tail ceiling based on its task contract. At checkback, inspect last meaningful progress, provider state and checkpoint before deciding to wait, pause, recover or stop. Keep valid long work alive while within its declared ceiling; escalate when the ceiling or a human-only gate is reached.
If a call may have completed but its acknowledgement is missing, reconcile the provider/job identifier and verifier state before replaying it. Retry without idempotency or reconciliation can duplicate work.
Keep failure classes distinct
WAIT_RESOURCE means capacity or provider limits; WAIT_AUTH means credentials need a separate recovery path. Neither should be inferred from a quiet heartbeat. Persist job state, wait for the specific dependency, then resume from the last verified boundary.
What the current observation establishes
In one reported incident, the user described waiting more than 60 minutes without a durable canary result. The exact earlier duration and in-flight outcome remain unknown. A later, separate bounded check used a 120-second deadline and succeeded; it did not resolve the earlier call or establish root cause. This is one scoped observation, not a detector benchmark.
Review checklist
- Name the signal and source for every status.
- Record semantic progress separately from heartbeat.
- Persist a checkpoint before a recoverable wait.
- Set a stage-specific checkback and ceiling.
- Reconcile ambiguous outcomes before retry.
- Finish only when the verifier accepts the artifact.
Use the AI Agent Operating Rules Builder, the incident observation, and the measurement method.