Harness Engineering & Sandboxes

A model call is not an engineered agent run. A model call accepts context and emits tokens; a run changes an environment over time. The moment generated output can read a repository, execute a command, call a service, or approve a transaction, the surrounding software becomes the primary safety and reliability boundary. Harness engineering is the discipline of designing that surrounding software so an attempted task is reproducible, constrained, observable, testable, and attributable.

The most useful mental model is an evidence envelope. The harness receives a versioned task specification, creates a known execution environment, grants explicit capabilities, records every consequential transition, and returns both artifacts and evidence. The artifact may be a patch, report, answer, or database update. Evidence explains how that artifact was produced: fixture identity, image digest, tool arguments, exit statuses, diffs, resource use, validation results, and terminal reason. A bare “success” string is not evidence. Neither is a persuasive model explanation. Success is a predicate independently computed from observable state.

This distinction separates a harness from a prompt wrapper. Prompt wrappers focus on instruction wording and model selection. Harnesses focus on contracts between untrusted or fallible decisions and real effects. They assume the planner may repeat an action, misunderstand a result, exceed a budget, or declare completion early. The harness therefore treats all proposed operations as requests. It validates the request, authorizes it against policy, executes it in a constrained adapter, bounds the returned observation, and appends a trace event. The model never receives ambient authority merely because it can name a tool.

A minimal run has six nouns. The task is the requested outcome and acceptance test. The fixture is the initial versioned state. The capability is a narrow permission such as “read files under workspace” or “send one request to this host.” The adapter is the trusted code that implements that capability. The observation is the typed bounded result returned to the planner. The verdict is the evaluator’s decision based on artifacts and trace evidence. Keeping these nouns separate prevents common category errors: a tool description is not authorization; a container is not a complete sandbox; a trace is not an evaluation; and an evaluator score is not permission to publish.

Consider a coding task. The task names a failing test and requires a patch. The fixture pins a repository commit and dependency lockfiles. The harness starts an isolated workspace with no production credentials, mounts the repository copy, and exposes read, edit, test, and diff tools. Each tool has an input schema, effect class, deadline, output cap, and audit event. The run terminates only when tests pass, the diff satisfies policy, and the budget remains valid. It can then emit a patch plus a manifest of the exact commands and environment. Another operator can replay the run from the fixture, even if a different model proposes the steps.

The design objective is not perfect determinism. Models, networks, clocks, and package registries can vary. The objective is controlled variance: identify what must be pinned, record what cannot be pinned, and distinguish deterministic checks from statistical outcomes. If an external dependency changes, the trace should make the change visible rather than allowing it to masquerade as model behavior.

Harness quality is measured by questions an incident reviewer can answer. What exact state entered the run? Which identity authorized each effect? Could repeated delivery duplicate a side effect? Where did untrusted text cross a trust boundary? Which limit stopped the run? Can a failing trajectory be replayed without production access? Which evaluator version produced the verdict? If the system cannot answer these questions, additional model intelligence will not repair the operational ambiguity.

The chapter sequence develops this evidence envelope from first principles. We begin with reproducible fixtures, then typed tools, isolation, runner lifecycle, traces and replay, budgets and recovery, fault injection, and finally evaluation and production operations. Each layer has a distinct responsibility. Defense in depth comes from composing them without pretending one layer—especially a container, prompt, or model refusal—is sufficient.