Chapter 5 of 8
Budget retries and detect loops that cannot converge
Chapter 5
About 5 minutes · includes examples, an exercise, and references
Chapter at a glance
- •Classify errors before retrying; ambiguous writes reconcile and permanent or policy failures do not repeat.
- •Propagate absolute deadlines, assign retry ownership, and preserve budgets across process restarts.
- •Combine capped exponential backoff and jitter with fleet-level load controls and circuit breaking.
Retries trade time and load for another chance at success. They do not repair invalid input, missing authority, violated invariants, or a committed effect whose response was lost. A retry policy begins with classification. Rejected requests require changed input or permission. Permanent failures end or compensate. Transient failures may improve with time. Throttling requires server guidance or paced demand. Ambiguous writes require reconciliation. Unknown failures are not automatically transient merely because retrying is easy.
Timeouts bound how long one operation occupies a resource; deadlines bound how long the caller is willing to wait for the larger goal. Pass one absolute deadline through nested work and derive shorter attempt timeouts from remaining time. Otherwise each layer can spend its full local timeout and the user-facing request lasts far longer than intended. Include connection establishment, DNS, TLS, queue delay, streaming, and response consumption in the budget you claim. Measure real latency distributions by dependency and operation.
Place retries at one layer that understands both semantics and budget. If an HTTP client retries three times, a service retries three times, and a workflow retries three times, one request can produce twenty-seven dependency attempts. The top layer often knows the business deadline, while the immediate caller knows whether the operation is idempotent and which errors are transient. Make the ownership decision explicit and expose lower-level attempt metadata upward.
Exponential backoff spaces repeated attempts, and jitter prevents many synchronized clients from retrying at the same moments. A typical delay is a random value bounded by a cap derived from base × 2^attempt. Honor Retry-After or equivalent service signals when trustworthy, but still enforce the caller's deadline. Backoff is not a capacity plan: unlimited queued retries can keep a failed dependency overloaded after it recovers. Use retry budgets, concurrency limits, token buckets, admission control, and circuit breakers.
A retry budget can be per run, tenant, dependency, and fleet. Per-run bounds stop one loop. Fleet bounds prevent a widespread failure from turning every caller into a load generator. Reserve capacity for fresh requests or critical reconciliation so retries do not starve new work. Record attempts consumed and next eligible time in durable state. A restarted worker must not reset the budget or discard the scheduled delay.
Circuit breakers suppress calls when recent failures indicate the destination is unhealthy. The open state fails or defers quickly; after a cooldown, a limited number of probes test recovery. Breakers protect resources but can create abrupt behavior and correlated probe storms, so combine them with jitter and bounded half-open concurrency. They do not replace timeouts, and they must not classify an ambiguous effect as absent.
Loops can fail while every individual call succeeds. Livelock occurs when state changes but useful progress does not: plan A produces error B, repair B recreates A, or two controllers continually undo each other. No-progress detection needs a semantic measure. Hash normalized state excluding timestamps and attempt counters; track unresolved-goal sets; count repeated state-action pairs; require a strictly decreasing rank for certain phases. A repeated tool call with equivalent arguments and no new evidence is a strong signal, but equivalence should ignore irrelevant serialization differences.
Oscillation is a structured form of no progress. Detect short cycles such as A→B→A using recent fingerprints. Add hysteresis, ownership, or a monotonic generation number rather than simply permitting more steps. When two reconcilers conflict, clarify which desired state is authoritative and use fencing or version checks. Randomness may hide a cycle in tests without fixing it.
Model-driven loops need special controls because a policy can narrate progress without changing operational state. Charge budgets when a proposal is made, not only when a tool succeeds. Require every continuation to identify new evidence, a changed hypothesis, or a reduced work item. Reject equivalent failed actions beyond a small threshold. Limit context growth; endlessly appending errors increases cost and may reduce decision quality. Summaries must preserve the state fields that no-progress detection uses.
Cancellation and expiry preempt retries. Check them before dispatch and after waking. A cancellation received during an ambiguous write does not prove the write was cancelled; move to reconciliation and stop unrelated work. Scheduled attempts should carry run generation and expected state version so stale timers cannot revive a completed or replaced loop. The handler must atomically verify that it still owns a legal transition.
Terminal behavior is part of retry policy. On exhaustion, record the last classified error, attempts and delays, authoritative effect status, remaining uncompensated work, and a recommended operator action. A high-risk loop may escalate early; a low-risk background derivation may schedule a long retry or mark degraded. The system should make a deliberate availability-versus-load-versus-risk choice rather than converting every failure into hidden latency.
Good retry engineering is observable restraint. It succeeds more often when failures are genuinely transient, reduces correlated load through paced attempts, stops when evidence says the strategy is ineffective, and leaves the workflow in a state from which a person or future event can safely continue.
Key points
- Classify errors before retrying; ambiguous writes reconcile and permanent or policy failures do not repeat.
- Propagate absolute deadlines, assign retry ownership, and preserve budgets across process restarts.
- Combine capped exponential backoff and jitter with fleet-level load controls and circuit breaking.
- Detect repeated semantic states, actions, and short cycles rather than trusting narrated progress.
Deadline-aware retries with full jitter and classification
Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.
from dataclasses import dataclass
from random import Random
class Transient(Exception): pass
class Permanent(Exception): pass
@dataclass
class FakeClock:
now: float = 0.0
def sleep(self, seconds: float) -> None:
self.now += seconds
def call_with_retry(
operation,
clock: FakeClock,
deadline: float,
attempts: int = 4,
attempt_timeout: float = 1.0,
) -> str:
rng = Random(7)
for attempt in range(attempts):
remaining_time = deadline - clock.now
if remaining_time <= 0:
break # never dispatch work after its absolute deadline
try:
result = operation(min(attempt_timeout, remaining_time))
except Permanent:
raise
except Transient:
if attempt + 1 == attempts:
break
if clock.now >= deadline:
break # the failed attempt's duration consumed the budget
cap = min(8.0, 0.5 * (2 ** attempt))
delay = rng.uniform(0, cap)
if clock.now + delay >= deadline:
break
clock.sleep(delay)
else:
if clock.now > deadline:
break # do not accept a result that arrived after the deadline
return result
raise TimeoutError("retry budget or deadline exhausted")
remaining = [(0.2, Transient()), (0.4, Transient()), (0.1, "ok")]
clock = FakeClock()
def flaky(timeout: float) -> str:
duration, value = remaining.pop(0)
clock.sleep(min(duration, timeout))
if duration > timeout:
raise Transient("attempt timed out")
if isinstance(value, Exception):
raise value
return value
assert call_with_retry(flaky, clock, deadline=5.0) == "ok"
assert 0 < clock.now < 5
print(round(clock.now, 3))Worked examples
Toy
Deterministic fault schedule
A dependency fails twice with a transient error and then succeeds.
A seeded random generator creates bounded jitter. The loop records each scheduled delay and succeeds on attempt three within its absolute deadline. A permanent-error variant stops after one attempt. The test asserts the maximum attempt count rather than waiting in real time.
- Classification precedes scheduling.
- All delay fits within the deadline.
- The test injects time instead of sleeping.
Application
Search-index stabilization
A newly written record may take seconds to appear in a search index.
Verification polls with jitter under a visibility deadline but queries the primary store before declaring failure. The index check is read-only and repeatable. If primary shows the record and index remains stale, the loop ends degraded and opens a repair task; it never repeats the original write.
- Visibility delay is distinct from write failure.
- Polling cannot duplicate the effect.
- Degraded terminal state preserves repair ownership.
System
Fleet-wide dependency outage
Thousands of workflows call the same service as its latency and error rate spike.
Admission limits new dependent work, a token bucket caps aggregate retries, and per-run backoff uses jitter. The breaker opens after measured failures and schedules limited probes. Reconciliation traffic for ambiguous writes has a reserved budget. Operators see queued obligations and can extend deadlines or degrade noncritical work.
- Retries cannot multiply without fleet bounds.
- Fresh and reconciliation traffic have explicit priority.
- Recovery probes are limited and jittered.
Exercise
Build a retry and no-progress policy
Select a dependency used by a long-running loop and design behavior under latency, throttling, outage, ambiguity, and logical cycling.
- Create an error taxonomy and map each class to fail, retry, reconcile, compensate, or escalate.
- Set absolute deadline, attempt timeout, capped backoff, jitter, and per-run and fleet budgets.
- Define semantic fingerprints, a progress measure, and a short-cycle rule.
- Specify cancellation, stale-timer, breaker, and exhaustion transitions.
Success criteria
- Nested layers cannot multiply attempts beyond the stated total.
- A restart preserves attempts and next eligible time.
- A repeated successful call that makes no useful progress is still stopped.
- Exhaustion leaves evidence and a safe continuation path.
Reflect: Which failure in this loop gets worse when every client retries independently?