Reading tools and contents
4 min
Harness Engineering & Sandboxes

Chapter 2 of 10

Reproducible fixtures and environment identity

Reproducibility

About 4 minutes · includes examples, an exercise, and references

Chapter at a glance

  • •Use immutable content identities, not moving tags.
  • •Separate declared dependency intent from attested build provenance.
  • •Treat external services, time, and randomness as explicit variable inputs.

A reproducible run starts before the first tool call. If the initial repository, dataset, dependencies, locale, clock, credentials, or service responses are unknown, a later trace can describe actions precisely and still fail to explain the result. A fixture is the complete declared starting state from which a task can be attempted. It should be addressable, immutable after publication, cheap to instantiate, and accompanied by enough provenance to rebuild or verify it.

Begin with content identity rather than names. A branch name, container tag, or “latest” dataset is a moving pointer. Record cryptographic digests for source archives, base images, lockfiles, model versions, evaluator definitions, and important input documents. Human-readable versions remain useful, but the digest is what lets two operators determine whether they used the same bytes. For a Git task, record the commit, submodule commits, dirty-state policy, and a hash of untracked fixture files. For a document task, record each source identifier, revision, extraction configuration, and normalized-content hash.

Environment identity has layers. The host kernel and architecture influence system calls and numerical behavior. The runtime image controls binaries and shared libraries. The language environment controls package resolution. Process settings control timezone, locale, random seeds, thread counts, and environment variables. External services introduce their own versions and data. Pin the layers that materially affect the acceptance test; record the rest. Claiming bit-for-bit reproducibility across heterogeneous accelerators may be unrealistic, while reproducing a failed schema validation should be exact.

Build provenance answers a different question from a lockfile. A lockfile states intended dependencies; provenance states which builder, inputs, parameters, and process produced an artifact. SLSA provenance and in-toto attestations provide reusable models for this evidence. A harness can emit an attestation for the execution image and another for the run artifact. The subjects are artifact digests. The predicate contains fixture identity, runner version, policy version, and evaluator version. Signing can provide tamper evidence, but signatures only establish who signed; they do not make a weak process trustworthy.

Hermeticity is a spectrum. A fully hermetic build obtains every input from declared content-addressed storage and denies undeclared network access. Many agent tasks cannot be fully hermetic because they intentionally query changing services. Split the run into zones. Provisioning may access approved registries and cache immutable artifacts. Core execution may have no network. A separate research adapter may access allow-listed sources and record response digests. This preserves meaningful replay without pretending the web is static.

Time deserves explicit treatment. Wall-clock time is often both input and evidence. Freeze or inject a logical clock for tests that should be deterministic. For tasks about current conditions, record the observation time and source revision. Never silently replace current data with a replay fixture; label replayed external responses so evaluators do not confuse historical evidence with present truth. The same rule applies to randomness. Seed pseudo-random components where appropriate and capture the seed, but recognize that remote model APIs may remain nondeterministic.

Fixture lifecycle should avoid shared mutable residue. Instantiate from a read-only base into a per-run copy-on-write layer. Assign a unique run directory and tenant identity. Verify no prior process, socket, cache entry, or credential remains. At teardown, capture declared outputs, hash them, then destroy the writable layer and revoke ephemeral credentials. Retention policy should distinguish small evidence manifests from potentially sensitive full filesystem snapshots.

The practical test for reproducibility is replay, not a long manifest. Select a sample of completed and failed runs, provision them on a clean worker, and rerun deterministic checks. Compare fixture digests, tool observations, final artifact hashes, and evaluator verdicts. Differences should be classified: intended model variance, expected environmental variance, fixture drift, adapter regression, or unexplained nondeterminism. Unexplained variance becomes a defect with an owner.

Avoid a common anti-pattern: packaging the current workstation into an image after the result is known. That preserves accidental state and cannot prove which inputs were necessary. Instead, define the fixture before execution, make missing dependencies fail early, and record any controlled mutation as an event. Reproducibility is strongest when the declared environment is smaller than the developer’s environment and when rebuilding it is a routine automated operation rather than an incident-only ritual.

Key points

  • Use immutable content identities, not moving tags.
  • Separate declared dependency intent from attested build provenance.
  • Treat external services, time, and randomness as explicit variable inputs.
  • Test replay regularly on clean workers.

Hash a fixture manifest deterministically

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

Hash a fixture manifest deterministicallypython
from __future__ import annotations
import hashlib, json
from pathlib import Path
from tempfile import TemporaryDirectory

def sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for block in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(block)
    return digest.hexdigest()

def fixture_manifest(root: Path) -> dict[str, str]:
    root = root.resolve(strict=True)
    entries = sorted(root.rglob("*"))
    if any(path.is_symlink() for path in entries):
        raise ValueError("fixture manifests reject symbolic links")
    files = [path for path in entries if path.is_file()]
    return {p.relative_to(root).as_posix(): sha256(p) for p in files}

with TemporaryDirectory() as directory:
    root = Path(directory)
    (root / "config.json").write_text('{"version":1}\n', encoding="utf-8")
    manifest = fixture_manifest(root)
    canonical = json.dumps(manifest, sort_keys=True, separators=(",", ":"))
    print(hashlib.sha256(canonical.encode()).hexdigest())

Exercise

Make a drifting task replayable

Audit a task that currently depends on a local checkout and live services.

  1. Inventory byte inputs, environment inputs, clocks, randomness, and remote responses.
  2. Choose which inputs to pin, record, stub, or intentionally leave variable.
  3. Design a clean-worker replay that classifies every difference.

Success criteria

  • Moving names are paired with immutable identities.
  • Network access is declared per execution phase.
  • Replay failure produces a diagnostic category rather than a generic mismatch.

Reflect: Which undeclared input is most likely to invalidate an evaluation result?

References and further reading