UK-based independent AI efficiency projectEvidence before claims · 2026

Token Optimisation

AI efficiency · measured
Discuss a diagnostic

A live heartbeat is not model progress: lessons from a stalled canary

Four separate tracks identify process liveness, provider activity, durable progress and verifier acceptance; none substitutes for another.

Written by

Token Optimisation

in

,

An automation wrapper can continue to respond while the operation a person cares about has stopped making observable progress. That distinction matters when a health check or canary is used to decide whether to wait, retry or recover.

Four Workflow SignalsUSER-REPORTED INCIDENT + SEPARATE CHECK. PROCESS ALIVE → PROVIDER ACTIVE → STAGE PROGRESS → VERIFIER RESULT. Reported wait >60 min; later 120-second check was separate and did not resolve earlier outcome.USER-REPORTED INCIDENT + SEPARATE CHECK1PROCESS ALIVE2PROVIDER ACTIVE3STAGE PROGRESS4VERIFIER RESULTReported wait >60 min; later 120-second check was separate and did not resolve earlier outcome.
USER-REPORTED INCIDENT + SEPARATE CHECK. Reported wait >60 min; later 120-second check was separate and did not resolve earlier outcome.

View the full infographic → · View this article

In one OpenCode recovery diagnostic, the last completed operation visible in the prior trace was a shared-backend health/start check. No later canary result was durable in that trace. The user reported that the turn had stalled for more than one hour; the exact duration and the in-flight call or exception could not be recovered. The event was recorded as elapsed >60 minutes with exact duration unavailable—not as a proven provider failure or success.

Missing result evidence does not show whether the request never started, remained in flight, completed without a durable callback, or failed after a response. Treating uncertainty as a known failure can trigger duplicate work; treating a live wrapper as success can hide a stall.

Four signals, four different questions

Signal Question answered What it does not prove
Process heartbeat Is a supervisor or worker responding? That a model request is progressing
Provider activity Did a request start, stream or complete? That the workflow accepted the response
Stage progress Did a named stage produce a durable artifact? That the artifact passed validation
Verification/acceptance Did defined checks pass on the artifact? That every operational dependency is healthy

Dashboards should display these separately. A heartbeat should not continually reset a deadline intended to detect missing task progress.

Recovery without blind replay

The recovery record used a bounded 120-second overall/no-progress deadline for a later validation attempt. It reused the already-live backend, did not restart it and did not replay the unobserved original canary. The uncertain earlier event and the later bounded check remained separate.

An uncertain stalled canary and a separate later successful check A health check completed, but no canary result was durable and a user reported more than 60 minutes of waiting; the exact outcome is unknown. A later separate bounded check succeeded. Keep uncertain attempts separate from recovery checks EARLIER ATTEMPT · OUTCOME UNKNOWN Health/start check completed No later canary result durable in visible trace. User reported >60 min Exact duration/call exception unavailable. reconcile; do not replay blindly LATER CHECK · SEPARATE EVENT Bounded 120-second check Reused the healthy backend; no restart. Canary completed successfully Does not resolve the earlier unknown outcome.
The user reported a delay greater than 60 minutes; the exact earlier outcome is unknown. The later bounded check is a separate event.
  1. Read the durable attempt/result record and identify the last completed operation.
  2. Check for a provider result, callback or artifact before issuing another request.
  3. If the outcome remains ambiguous, record UNKNOWN rather than inventing success or failure.
  4. Apply an explicit overall and no-progress deadline to a new bounded attempt.
  5. Reuse a healthy backend only when evidence supports it; restart only when diagnostics justify it and ownership permits it.
  6. Verify the new result independently and keep it distinct from the unresolved prior attempt.

What the bounded follow-up established

A later, separate OpenCode-managed OAuth canary requested openai/gpt-6-luna, returned exactly KD_OPENCODE_AUTH_OK, exited successfully, and reported 13,289 input tokens, zero cached input tokens, 10 output tokens and 13,299 total tokens.

This establishes that the later bounded canary completed in that configured environment. It does not retroactively resolve the earlier call, explain the stall, prove general provider reliability, or measure task quality.

Measure useful progress

  • Record stable job and attempt IDs, provider/model route and timestamps.
  • Track heartbeat, provider activity, semantic progress and terminal state separately.
  • Preserve durable artifact path/hash and the verifier checks performed.
  • Record timeout classification and whether replay was suppressed pending reconciliation.

Do not infer backend ownership from a matching port or process alone. A live wrapper is not proof that its task stage is advancing.

Limitations

The original stall duration is only known to exceed one hour from a user report; its exact value is unavailable. The in-flight request and exception were not recoverable from the visible trace. The later successful canary is a separate validation. These observations do not support a two-hour duration or a root-cause conclusion.

Further reading and tools

Diagnostic enquiry

Teams can enquire about reviewing observable progress and token accounting boundaries in a recurring AI workflow through the Token Optimisation contact route. No outcome or savings are guaranteed; agree data scope before sharing workflow material.

More posts