Make history replayable and upgrades explainable

A loop that must survive process failure needs durable history. Saving only the latest phase is often insufficient: operators cannot reconstruct why it changed, late events are hard to classify, and upgraded code may reinterpret old data. An event history records the accepted facts that drove state transitions. Current state is a projection obtained by reducing those facts. A checkpoint stores a validated projection at a known history position so recovery need not replay from the beginning.

An event envelope should include a stable event identifier, run identifier, aggregate or workflow identifier, sequence or expected version, event type and schema version, recorded time, source time when relevant, actor or producer, causation identifier, correlation or trace identifiers, and a payload reference or bounded payload. Identity supports deduplication. Sequence detects gaps and conflicting writers. Causation explains which command produced a result. Schema version makes evolution explicit. Do not rely on wall-clock timestamps alone to order concurrent events.

Persist decisions before effects when losing the intent would be unsafe. A common pattern commits the next state and an outbox command atomically. A dispatcher later sends the command and records dispatch metadata; the result returns as another event. This does not create exactly-once networks. It ensures the controller can rediscover intended work after a crash. The receiver still needs idempotency, and the controller still needs reconciliation when delivery outcome is ambiguous.

Replay requires deterministic workflow logic. Given the same accepted history and compatible code version, rebuilding state should produce the same commands at the same logical points. Wall clocks, random values, environment reads, and model outputs are nondeterministic; record their results as events or obtain them through runtime APIs that persist a marker. A workflow should not call a payment API during replay. It should encounter the recorded payment result and reconstruct state without repeating the effect.

Temporal's durable execution model uses event history to resume workflow code and imposes determinism constraints because changed workflow code can make different decisions for an existing history. The general lesson applies outside any engine: long-running instances outlive deployments. Version transition logic. Keep old handlers while old histories exist, use explicit version markers for branch changes, migrate state through reviewed transformations, or start a new generation with a recorded handoff. “Deploy and hope replay matches” is not an upgrade strategy.

Checkpoints trade replay cost for storage and validation complexity. A checkpoint needs aggregate identifier, history position, state-schema version, projection digest, creation code version, and integrity protection. On recovery, load the latest compatible checkpoint, verify it, and replay subsequent events. Periodically rebuild from the full history and compare digests to detect projection bugs. Never delete the only audit evidence merely because a checkpoint exists; retention depends on business, privacy, and incident requirements.

Events can arrive more than once, late, or out of order. Deduplicate by stable event identity at the aggregate boundary. Enforce expected version for transitions that require ordering. For commutative facts, a set or monotonic merge may accept reordering. For noncommutative facts, store the late event as observed but do not apply it silently; route it to reconciliation or an operator. The distinction between recorded and applied events keeps the audit log honest.

Large payloads should be immutable artifacts addressed by content digest or stable version, while history stores metadata and references. This avoids enormous logs and ties approvals, evaluations, and effects to exact bytes. Verify artifact integrity when loading. Apply access control and retention to both events and artifacts; event sourcing is not permission to preserve personal data forever.

Replay is also a debugging and evaluation tool. An operator can rebuild state before a defect, inspect the decisive event, fork a simulation with corrected logic, and compare outcomes without executing real effects. A team can replay production-shaped histories through a candidate reducer and check invariants before deployment. Because model outputs and tool observations are recorded, the control logic can be tested independently from their cost and variance.

There are two kinds of replay. State replay reconstructs what the system believed at a point in history. Decision replay reruns new logic against recorded external inputs to study what it would have decided. The latter is counterfactual and must be clearly labelled; it does not replace the historical record. Never overwrite the past with a more convenient interpretation.

A well-engineered history lets the team answer: What did the loop know? Which code and policy interpreted it? Which command was intended? Was it dispatched? What evidence confirmed the result? Did an upgrade alter the path? If those questions require searching unstructured logs across services, persistence captured symptoms but not the workflow.