Chapter 8 of 8

Operate loops with traceable authority and safe controls

Chapter 8

About 5 minutes · includes examples, an exercise, and references

Chapter at a glance

•Propagate interoperable trace context and instrument lifecycle transitions, effects, waits, joins, and verification.
•Measure stable outcomes, unresolved obligations, invariant health, and resource use by meaningful cohort.
•Keep histories and telemetry privacy-aware while recording the authority chain for consequential effects.

An operational loop must answer three questions in real time: Is useful work progressing? Are invariants and authority intact? What obligation remains if the run stops now? Observability is the instrumentation that makes these questions measurable. Operations is the ownership, policy, and response system that acts on the answers. A large volume of logs without lifecycle semantics provides neither.

Start with a canonical run identity and propagate trace context across model calls, queue messages, workers, tools, child workflows, and external adapters. W3C Trace Context standardizes traceparent and tracestate propagation; OpenTelemetry defines traces, spans, events, links, attributes, status, and context behavior. A trace follows causal work, but durable and asynchronous loops may need span links when one event relates to several prior contexts or when work resumes long after the parent span ended.

Choose spans around operational boundaries: admission, policy decision, state transition, model inference, tool dispatch, external request, reconciliation, checkpoint, human wait, child join, and verification. Attach bounded attributes such as run type, state phase, transition name, tool and policy version, attempt, terminal reason, and risk tier. Never attach raw secrets, full private documents, or unrestricted model context as attributes. High-cardinality identifiers are useful for investigation but need storage and access planning.

Events explain discrete facts inside a span: guard_denied, retry_scheduled, approval_consumed, stale_timer_ignored, no_progress_detected, circuit_opened, or compensation_started. Logs should be structured and carry run, trace, event, command, and effect identifiers. Metrics aggregate behavior: admitted runs, verified completions, invariant violations, terminal reasons, active waits, oldest obligation age, retries, ambiguous effects, reconciliation latency, compensation backlog, cost, and queue lag.

Measure end-to-end usefulness, not component vanity. A 99.9% tool-call success rate can coexist with low verified completion if tools are called unnecessarily. Track verified completion rate, correct escalation, p50 and p95 time to stable state, cost per verified completion, steps per terminal reason, no-progress stops, and durable-effect ambiguity. Slice by workflow type, risk tier, tenant class, policy version, model, tool version, and dependency so an average does not hide a failing cohort.

Service objectives should reflect obligations. A synchronous assistant may target response latency; a durable loop may target time to acknowledgement, percentage reaching a stable state within a window, and maximum age of an unresolved external effect. Define an error budget for failed or late verified outcomes, but treat critical invariant violations as separate stop-ship events rather than spending them like ordinary latency errors. Alert on symptoms that require action: growing ambiguous-effect age, stuck-state cohorts, trace gaps, write-after-cancel attempts, fencing rejection spikes, and exhausted reconciliation.

Security telemetry follows authority. Record which authenticated principal initiated a run, which policy granted a capability, the resource scope, approval digest and consumer, credential issuance reference, tool version, and effect receipt. Keep sensitive values out of model-visible state when unnecessary. Separate operator, auditor, and developer access to traces. Apply tenant isolation, encryption, retention, deletion, and export rules to histories and observability stores; they often contain derived personal or proprietary data.

Operational controls must sit outside the loop's policy. Provide admission pause, per-tool disable, write-safe mode, tenant quarantine, model or prompt rollback, credential revocation, queue drain, run cancellation, and reconciliation-only mode. A kill switch that depends on the same compromised model choosing to honor a sentence is not a control. Test controls regularly and record activation as auditable events.

Runbooks map alerts to safe actions. For an ambiguous payment backlog: stop new payment dispatches if risk warrants, preserve reconciliation reads, inspect provider health, sample effect identities, reconcile authoritative status, resume with rate limits, and communicate outstanding obligations. For a replay divergence: halt affected deployments, preserve histories, identify the behavior version, use a compatible worker, and validate a migration before resuming. Each runbook names an owner and escalation deadline.

Capacity planning includes more than active workers. Estimate event history growth, checkpoint load, timer cardinality, queue backlog, external rate limits, model tokens, artifact storage, trace volume, and operator review. Apply backpressure at admission so overload becomes explicit queued or rejected work instead of unbounded latency. Reserve capacity for cancellation, reconciliation, and compensation because these reduce risk during incidents.

Change management treats workflow logic, state and event schemas, policies, model configurations, tool adapters, and sandbox images as independently versioned release artifacts. Verify historical replay, fault tests, migration, dashboards, alerts, and rollback before rollout. Use canaries by tenant or risk tier; shadow model decisions without effects; expand authority only after evidence. Record exact versions on every run so an incident cohort can be found.

Post-incident review reconstructs causal history from trace and event records, then focuses on failed assumptions and controls. Reconcile all external effects first. Identify whether detection was late, containment incomplete, recovery unsafe, or runbooks unclear. Add a minimized history to regression tests, update invariants or budgets, and verify the remedy with fault injection. Do not claim closure while an external obligation remains unknown.

The final maturity test is graceful degradation. When a model, tool, queue, or policy service fails, the loop stops expanding risk, preserves accepted work, exposes its exact state, and retains a bounded route to resume, reconcile, compensate, or escalate. Observability makes that state legible; security limits what failure can reach; operations gives a person the authority and evidence to finish the job.

Key points

Propagate interoperable trace context and instrument lifecycle transitions, effects, waits, joins, and verification.
Measure stable outcomes, unresolved obligations, invariant health, and resource use by meaningful cohort.
Keep histories and telemetry privacy-aware while recording the authority chain for consequential effects.
Provide externally enforced containment, reconciliation, rollback, and runbooks, then test them under fault.

Structured loop events with propagated trace identity

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

Structured loop events with propagated trace identitypython

from __future__ import annotations

from dataclasses import dataclass, asdict
from datetime import datetime, timezone
import json, uuid

@dataclass(frozen=True)
class LoopEvent:
    trace_id: str
    run_id: str
    phase: str
    transition: str
    attempt: int
    terminal_reason: str | None = None

def emit(event: LoopEvent) -> None:
    record = asdict(event)
    record["recorded_at"] = datetime.now(timezone.utc).isoformat()
    # Payloads, prompts, credentials, and customer data are deliberately absent.
    print(json.dumps(record, sort_keys=True))

trace_id = uuid.uuid4().hex
run_id = "run-104"
emit(LoopEvent(trace_id, run_id, "admitted", "admit", 0))
emit(LoopEvent(trace_id, run_id, "reading", "retry_scheduled", 1))
emit(LoopEvent(trace_id, run_id, "verifying", "effect_confirmed", 2))
emit(LoopEvent(trace_id, run_id, "succeeded", "finish", 2, "verified_success"))

Worked examples

Toy

Trace one retrying loop

A three-step loop makes a transiently failing read before reaching success.

One run trace contains admission, two read-attempt spans, retry_scheduled, state-transition, and verifier spans. Both attempts share the run and trace identity; attempt number and error class are attributes. The completion metric increments only after the verifier event.

Retry attempts remain causally connected.
The error value is bounded and non-sensitive.
Success derives from verification, not span status alone.

Application

Ambiguous-refund operations panel

Support operators need to manage refunds whose provider responses were lost.

The panel groups obligations by age, provider, risk tier, and reconciliation status. Each item links canonical intent, approval, dispatch attempt, trace, and provider query receipts. Operators can reconcile or escalate but cannot create a second key. An alert fires on oldest age rather than raw timeout count.

The dashboard centers unresolved obligations.
Operator actions preserve original intent identity.
Access excludes unrelated customer context.

System

Safe mode during a policy regression

A new policy version unexpectedly authorizes writes outside one tenant scope.

An invariant alert identifies the policy cohort. External controls disable write capabilities for that version and revoke issued run credentials while allowing read-only reconciliation. Exact version attributes locate affected runs and receipts. The team rolls back, reconciles effects, adds the trace to regression tests, and canaries the correction.

Containment does not depend on workflow cooperation.
Versioned telemetry defines the incident cohort.
Recovery accounts for already-dispatched effects.

Exercise

Create an operations contract for a durable loop

Design instrumentation, objectives, containment, and response for a loop that owns an external obligation.

Define trace boundaries, propagation, event names, safe attributes, metrics, and privacy rules.
Choose verified-outcome and unresolved-obligation objectives plus invariant alerts and diagnostic slices.
Specify externally enforced safe mode, tool disable, credential revocation, cancellation, and rollback.
Write one incident runbook and a post-incident regression and reconciliation checklist.

Success criteria

An operator can reconstruct authorization, transition, dispatch, and confirmation from stable identities.
Alerts point to an actionable obligation or invariant rather than generic log volume.
Telemetry does not expose raw secrets or unrestricted private context.
Containment preserves the ability to reconcile and compensate existing effects.

Reflect: Which obligation could remain unknown today even while every service dashboard appears healthy?