Chapter 6 of 10
Traces, event records, and replay semantics
Observability
About 4 minutes · includes examples, an exercise, and references
Chapter at a glance
- •Separate the durable recovery journal, distributed traces, logs, and large evidence artifacts.
- •Trace context expresses causality, not identity or authorization.
- •Define replay mode explicitly and never repeat production effects by default.
A harness trace is an evidence graph, not a transcript dump. Conversational text can help a developer, but reliable investigation depends on structured events with stable identities and causal links. Each event should answer who requested what, under which policy and budget, what effect occurred, what observation returned, and which state transition was committed. Trace design must balance reconstruction value against cost, privacy, and exposure of sensitive model context.
Use three related records. The durable run journal contains ordered state transitions and is required for recovery. The distributed trace links work across orchestrator, worker, tool services, evaluators, and artifact stores. Logs provide detailed diagnostics local to a component. Do not assume one signal substitutes for the others. A sampled distributed trace may be unavailable during recovery; the run journal must remain complete enough to resume safely. Logs may be verbose and mutable; critical verdict evidence belongs in immutable artifacts or journal entries.
Represent each model request, tool validation, authorization decision, adapter execution, checkpoint, evaluation, and export as a span or event. Attach low-cardinality attributes suitable for metrics: task class, model identifier, tool name, result class, policy version, evaluator version, and tenant-safe identifiers. Store large prompts, outputs, diffs, and test reports as content-addressed artifacts, then place redacted digests and references in the trace. High-cardinality or sensitive values can make telemetry expensive and dangerous.
Trace context propagation must not become authorization. W3C traceparent connects causal work across services; it says nothing about caller identity or permission. Validate incoming context and consider restarting it at trust boundaries. Never place secrets or personal data in tracestate. Generate separate authenticated request identities and carry them through service-to-service credentials or signed claims.
Replay has several meanings. Presentation replay renders the original events for a human. Evaluator replay reruns new scoring logic against preserved artifacts without re-executing effects. Simulation replay feeds recorded observations to a planner to compare policies. Full execution replay provisions the fixture and repeats tool effects in an isolated environment. These modes have different safety and fidelity. A production side effect should not be repeated merely because someone clicked replay.
Deterministic simulation requires the planner to receive the same normalized observations at the same points. Record adapter contract versions, truncation decisions, error classes, and ordering. If the original run depended on a live service, use a recorded response fixture and label the replay. The result estimates how the planner reacts to preserved evidence, not how the live service behaves today. Counterfactual replay can compare two models on the same trajectory, but it becomes invalid after their actions would have changed later observations; branch the simulation at that point.
Sampling policy should reflect consequence, not only latency. Retain complete evidence for policy denials, ambiguous effects, evaluator disagreement, security detections, and high-consequence actions. Sample routine successful detail more aggressively while keeping aggregate metrics. Tail sampling can preserve traces after their outcome is known, but the durable journal should not depend on an exporter’s eventual decision. Define retention windows and access controls separately for traces, artifacts, and customer content.
Redaction must occur before untrusted or broadly accessible sinks. Maintain an allow-list of trace attributes and structured redactors for tool schemas. Hashing a secret is often unsafe if the input space is small. Use opaque identifiers or keyed digests when correlation is required. Record that redaction occurred and which policy version applied, so missing data is explainable during an incident.
Test trace completeness. Create contract tests that assert every committed effect has a preceding authorization event, an idempotency identity, a bounded observation or explicit lost-observation marker, and a resource charge. Assert terminal success links to evaluator artifacts and final artifact digests. Run these tests under cancellation, worker crash, exporter failure, and output truncation.
The goal is not maximal telemetry. The goal is sufficient, trustworthy evidence for recovery, evaluation, debugging, accountability, and improvement. A smaller typed trace whose semantics are stable is more valuable than gigabytes of interleaved text.
Key points
- Separate the durable recovery journal, distributed traces, logs, and large evidence artifacts.
- Trace context expresses causality, not identity or authorization.
- Define replay mode explicitly and never repeat production effects by default.
- Redact before export and test evidence completeness under failure.
Append a hash-chained run event
Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.
import hashlib, json, time
def append_event(events: list[dict], kind: str, payload: dict) -> dict:
previous = events[-1]["event_hash"] if events else "0" * 64
body = {
"sequence": len(events),
"time_ns": time.time_ns(),
"kind": kind,
"payload": payload,
"previous_hash": previous,
}
encoded = json.dumps(body, sort_keys=True, separators=(",", ":")).encode()
event = {**body, "event_hash": hashlib.sha256(encoded).hexdigest()}
events.append(event)
return event
journal: list[dict] = []
print(append_event(journal, "tool.authorized", {"tool": "read_file"}))Exercise
Specify a safe replay
An evaluator regression requires rerunning scores for one thousand historical agent runs.
- Choose evaluator, simulation, or execution replay and justify the boundary.
- List preserved artifacts and event fields required for fidelity.
- Define privacy, retention, and access-control constraints.
Success criteria
- Replay cannot repeat external side effects.
- New evaluator identity remains distinct from the original verdict.
- Unavailable or redacted evidence produces an explicit limitation.
Reflect: Which question can your current logs not answer without reconstructing state from prose?