Chapter 6 of 8

Coordinate durable work without hiding concurrency

Chapter 6

About 5 minutes · includes examples, an exercise, and references

Chapter at a glance

•Treat pauses, timers, signals, child work, and joins as durable event-driven transitions.
•Name fork and join semantics, cancel losing branches, and reconcile any late effects.
•Use optimistic versions, leases with fencing, and generation checks to reject stale owners and timers.

A durable workflow is a loop whose state and future obligations survive worker loss, process restart, and deployment. Durability changes the programming model. A function call that appears sequential may pause for hours, resume on another machine, receive duplicate events, or run under newer infrastructure while preserving older workflow semantics. Design around durable state and explicit events rather than thread ownership.

Control-flow patterns provide a vocabulary for composition. Sequence runs B after A. Parallel split starts independent branches. Synchronization waits for all required branches. Exclusive choice selects one branch. Multi-choice selects several. Deferred choice lets the first qualifying external event decide. Milestone enables an action only while another state holds. Cancellation region stops a defined set of activities. Naming the pattern exposes join semantics and failure behavior that a generic “parallel tasks” label hides.

For every fork, define the join. Does the parent need all results, the first success, a quorum, the best result before a deadline, or any terminal result? What happens to late branches? Are their effects still legal? A race that returns the first answer but leaves other writers running is not complete. Propagate cancellation, fence stale results, and record which branch satisfied the join. If a losing branch may have created an effect, reconcile or compensate it.

Signals and messages change a waiting workflow. Validate identity, schema, expected state, generation, and deduplication identifier before applying them. A human approval that arrives after expiry is a late fact, not permission to reopen the old execution. Record it for audit and reject its transition. Query operations should be read-only projections; commands that change workflow state should enter through the same event and guard path as any other transition.

Concurrency requires an ownership rule. Optimistic concurrency uses an expected state version: only one writer commits, while losers reload and reconsider. Leases grant temporary ownership, but a paused worker can continue after its lease expires. Add a monotonic fencing token to every external write; the receiver rejects commands from older owners. A distributed lock without fencing can protect the database while failing to protect the external service.

Avoid holding locks while calling slow or human systems. Commit an intent, release the transaction, perform the effect, then apply the result with expected version and command identity. If another event changes state meanwhile, the result handler checks whether the result is still relevant. The result may be ignored, reconciled, compensated, or applied to a newer compatible state. Do not force the world to behave like one database transaction.

Child workflows isolate lifecycle and scale. Give each child a stable identity, input contract, authority set, deadline, and terminal result. The parent records the child start before dispatch and consumes completion idempotently. Decide whether child cancellation follows parent cancellation, whether a child can outlive the parent to finish compensation, and how version upgrades are coordinated. Children are not merely function calls with nicer dashboards.

Long-running work needs timers as durable events. Store wake-up time and generation, then deliver timer_fired at least once. The reducer verifies the run is still in the expected waiting state and that the generation matches. This makes duplicate or late timers harmless. Use calendar semantics carefully: “24 hours” and “next business day at 09:00 in the customer's zone” are different contracts with daylight-saving and holiday implications.

Fan-out can overload dependencies and history. Bound concurrency, batch when semantics allow, stream results into an incremental join, and apply backpressure. Store large child outputs as artifacts rather than embedding them in every parent event. Define whether one branch failure fails all, reduces quorum, triggers replacement, or yields partial success. A partial-success terminal state must identify completed and outstanding obligations.

Kubernetes reconciliation demonstrates multiple controllers cooperating through declared resource state and ownership. The lesson for workflows is to avoid two controllers silently writing the same field. Partition ownership, use generation and observed-generation fields, and publish conditions rather than overwriting another component's conclusion. When multiple desired-state authors exist, specify precedence or merge rules.

Durable engines can provide history, timers, activity retries, and recovery, but they cannot choose business invariants. Workflow code still needs idempotent activities, bounded histories, versioning, cancellation semantics, security boundaries, and operator states. Treat engine guarantees precisely: know whether a worker task, activity invocation, signal, or external effect is delivered at least once and where deduplication occurs.

The design test is failure at every await. Stop the worker before dispatch, after dispatch, after an effect commits, during a join, after cancellation, and while code is upgraded. For each point, the durable record should cause exactly one safe next decision. If correctness depends on the original stack frame returning, the workflow is long-lived in duration but not durable in design.

Key points

Treat pauses, timers, signals, child work, and joins as durable event-driven transitions.
Name fork and join semantics, cancel losing branches, and reconcile any late effects.
Use optimistic versions, leases with fencing, and generation checks to reject stale owners and timers.
Know the engine guarantee at every boundary; idempotency and business invariants remain application responsibilities.

A generation-safe first-success join

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

A generation-safe first-success joinpython

from __future__ import annotations

from dataclasses import dataclass, field
from threading import Lock
from typing import Callable

@dataclass
class Join:
    generation: int
    branches: frozenset[str]
    winner: str | None = None
    seen: set[str] = field(default_factory=set)
    lock: Lock = field(default_factory=Lock, repr=False)

Oracle = Callable[[dict], bool]

def accept_result(join: Join, event: dict, oracle: Oracle) -> tuple[Join, list[str]]:
    if set(event) != {"event_id", "generation", "branch", "payload"}:
        return join, []
    if (
        not isinstance(event["event_id"], str)
        or not event["event_id"]
        or event["branch"] not in join.branches
        or not isinstance(event["payload"], dict)
    ):
        return join, []
    passes = oracle(event["payload"])  # expected truth is not supplied by the branch
    with join.lock:  # one transaction or compare-and-set in a durable store
        if event["event_id"] in join.seen:
            return join, []
        join.seen.add(event["event_id"])
        if event["generation"] != join.generation or join.winner is not None or not passes:
            return join, []
        join.winner = event["branch"]
        losers = sorted(join.branches - {join.winner})
        return join, [f"cancel:{name}" for name in losers]

trusted_answer = 42
def answer_oracle(payload: dict) -> bool:
    return payload.get("answer") == trusted_answer

join = Join(generation=3, branches=frozenset({"a", "b"}))
event = {"event_id": "e-1", "generation": 3, "branch": "b",
         "payload": {"answer": 42}}
join, commands = accept_result(join, event, answer_oracle)
assert join.winner == "b" and commands == ["cancel:a"]
join, duplicate_commands = accept_result(join, event, answer_oracle)
assert duplicate_commands == []
print(join)

Worked examples

Toy

First-success race

Two read-only solvers run concurrently and the first verified answer wins.

Each branch has an identity and result event. The join accepts the first result that passes a deterministic verifier, records the winner, and cancels the other branch. A late result is stored but cannot replace the winner because the join generation is closed.

Join condition is verified first success, not first response.
Late branch events are idempotent.
Read-only losers require no effect compensation.

Application

Quorum document extraction

A workflow runs three independent extractors and needs two matching totals before a payment review.

Branches run under a concurrency bound and store artifact digests. The incremental join normalizes totals and records a quorum when two match. A deadline with no quorum produces needs_review with all evidence. No extractor can authorize payment; the join only prepares an observation.

Quorum semantics are explicit.
Large results live in immutable artifacts.
Deadline produces partial evidence, not invented consensus.

System

Fenced deployment controller

A worker pauses during rollout, loses its lease, and later resumes while a new worker owns the same deployment.

The new owner receives a higher fencing token. Every command to the deployment service carries the token, which rejects the stale worker's lower value. State commits use expected version. Observed generation confirms which desired revision reached the environment.

Lease expiry alone is not trusted.
External writes enforce fencing.
Desired and observed generations are distinct.

Exercise

Design a durable concurrent workflow

Choose a workflow with parallel work, a human or timer wait, and an external effect, then specify behavior across restarts and races.

Name control-flow patterns and define every fork, join, partial result, and losing branch.
Specify event identities, expected versions, timer generations, lease or fencing requirements, and child contracts.
Walk through cancellation and completion arriving in both orders.
Inject a crash before and after each durable await and state the recovery transition.

Success criteria

No branch continues with write authority after it loses ownership.
Duplicate and late signals or timers cannot reopen closed generations.
Every join explains how incomplete, failed, and late branches are handled.
The workflow can resume on a different worker without an in-memory prerequisite.

Reflect: Which concurrent branch can still create an effect after the parent believes it has finished?