Runner architecture and lifecycle state machine

A production harness is easiest to reason about as a state machine whose transitions are committed by trusted code. “Running” is too coarse. A useful lifecycle includes accepted, validated, queued, provisioning, ready, executing, evaluating, exporting, terminal, and cleanup states. Failure can occur in every state, and cleanup must remain reachable from each one. The state record is durable; worker processes are replaceable executors.

Acceptance authenticates the caller and assigns a run identifier before expensive work. Validation resolves the task class, fixture, policy, budgets, evaluator, and requested capabilities. No environment is created until the run specification passes. The validated specification becomes immutable or versioned; later approvals produce explicit amendments rather than mutating history. Queueing should carry tenant and priority information without granting worker authority.

Provisioning materializes the fixture and isolation boundary. The worker verifies image and input digests, creates the writable layer, applies runtime policy, injects ephemeral credentials, and performs readiness checks. Provisioning failure is not a model failure. Keep infrastructure errors separate from task verdicts so model comparisons are not contaminated by unhealthy workers. Once ready, record the exact environment identity and start the execution budget clock.

Execution is a sequence of authorized adapter invocations. The orchestrator owns the loop: load state, ask the planner for a proposed transition, validate and authorize it, execute at most the allowed effect, append observation and resource charge, checkpoint, then evaluate terminal conditions. Never let arbitrary model output call worker APIs directly. A worker should accept a signed run specification and narrow adapter requests from an authenticated orchestrator.

Evaluation runs in a distinct context. For code, mount final artifacts read-only into a fresh evaluator environment so the task process cannot tamper with test binaries or reports. For retrieval or agent tasks, preserve query fixtures and grader versions. Separate deterministic invariants from statistical or model-graded criteria. A deterministic policy violation can fail the run even when a semantic evaluator likes the answer.

Export copies only declared artifacts through a validating broker. It computes digests, content types, sizes, and malware or policy scans as appropriate. The artifact manifest is committed before a terminal success state. If export fails after task success, the run is not safely deliverable; retain a recoverable state or report a distinct export failure. Terminal reasons should be enumerable: verified success, task failure, invalid request, policy denial, budget exhausted, cancelled, infrastructure failure, evaluator failure, or cleanup failure.

Cleanup is a first-class state rather than a finally block hidden in one process. Revoke credentials, cancel child operations, kill the resource domain, detach networking, capture allowed diagnostics, destroy writable storage, and release leases. Each step should be idempotent because workers can crash mid-cleanup. A reconciliation controller scans nonterminal and terminal-but-not-cleaned runs, checks external resources, and resumes cleanup without depending on the original worker.

Leases prevent duplicate workers from executing the same run concurrently. The durable run record stores a lease owner and expiration. Long operations renew the lease; expired work can be adopted after reconciliation. Adoption does not blindly repeat effects. It loads the last committed transition, checks idempotency records and external state, then resumes from a safe boundary. Exactly-once execution is rarely available across distributed systems; design for at-least-once delivery with idempotent commits and explicit ambiguity.

Cancellation also needs semantics. A caller can request cancellation, but the harness decides when and how to stop. Mark cancellation requested durably, stop issuing new effects, attempt cooperative cancellation for bounded time, then enforce termination. Evaluation may still run on partial artifacts if policy requires evidence. The terminal record must distinguish user cancellation from timeout or policy intervention.

The lifecycle state machine provides operational leverage. Metrics can count time in each state, reveal provisioning bottlenecks, separate infrastructure and task failures, and detect cleanup debt. Incident responders can see the last committed transition rather than infer it from interleaved logs. Most importantly, the state machine prevents the model’s conversational narrative from becoming the system of record.