Operating a harness as security-critical infrastructure

A production harness is a privileged execution platform. It schedules untrusted decisions, provisions compute, brokers credentials, reaches data, and decides which artifacts leave isolation. Operational maturity therefore matters as much as runner code. Ownership, deployment, capacity, incident response, policy change, and evidence governance need explicit design.

Split the system into trust zones. An edge authenticates callers and applies request quotas. A control plane validates run specifications, policy, approvals, and budgets. A scheduler assigns work without exposing broad credentials. Workers provision isolation and execute narrow adapters. Artifact and telemetry services accept content through authenticated, size-bounded channels. Evaluators operate on read-only evidence. Each service identity should have least privilege, and no single task worker should possess control-plane administration.

Use immutable versioned deployments for runner, adapters, policies, fixtures, and evaluators. A run records all versions. Roll out through shadowing and canaries stratified by task class and consequence. Compare candidate and baseline on success, policy denial, infrastructure failure, cleanup latency, cost, and security signals. Provide rapid rollback without rewriting historical evidence.

Capacity planning is multidimensional. CPU and memory are obvious, but cold-start slots, image pulls, artifact bandwidth, external API quotas, evaluator capacity, trace storage, and cleanup backlog can bottleneck. Apply admission control before accepting work that cannot meet deadline. Per-tenant fairness prevents one looping workload from occupying the fleet. Backpressure should surface as typed queue or capacity status rather than causing hidden timeouts.

Define service-level objectives around user outcomes and control health: accepted-to-start latency, verified completion, infrastructure failure rate, budget-overrun count, cleanup completion time, lost-observation count, duplicate-effect count, and evidence export success. A high task success rate can conceal unsafe cleanup or chronic evaluator outages. Page on symptoms that threaten invariants, not every individual model error.

Incident response begins with containment. Be able to disable a tool version, revoke a capability class, block an image digest, deny a destination, stop new high-risk runs, and terminate affected workers. Preserve evidence before destructive cleanup when policy permits. Maintain run-to-tenant and artifact-to-run indices so responders can scope impact. If trace context crosses trust boundaries, do not rely on it as tenant identity.

Secrets should be short-lived and capability-scoped. Use a broker to mint task credentials after authorization and revoke them at cleanup or cancellation. Prevent credentials from appearing in prompts, observations, logs, shell history, core dumps, and artifacts. Rotate broker keys and test revocation. A sandbox limits where secrets can travel only if network and artifact paths are also controlled.

Supply-chain controls apply to base images, workers, adapters, and evaluators. Verify image signatures or attestations according to organizational policy, pin digests, scan dependencies, and record provenance. Avoid installing arbitrary packages from the public internet during privileged execution. If tasks need dependencies, resolve them through a controlled cache during a separate phase and include identities in the fixture.

Evidence governance defines retention and access. Full prompts and files may contain customer secrets or personal data; traces may reveal system topology; artifacts may contain generated malware. Classify each evidence type, encrypt it, enforce tenant-scoped access, log reads, and delete according to policy. Keep the minimum durable recovery journal even if verbose content is sampled or short-lived.

Run operational game days. Exercise worker compromise, policy-service outage, leaked task credential, artifact-store unavailability, evaluator regression, and runaway cost. Verify kill switches and degraded modes. Review whether responders can distinguish infrastructure, model, policy, and customer-caused failures. Feed lessons into fixtures, fault tests, and runbooks.

Mastery is demonstrated by a complete evidence chain. A defensible capstone provisions an immutable fixture, validates typed tools, executes inside a threat-modelled boundary, enforces budgets, records a replayable journal and distributed trace, injects controlled faults, evaluates independent success, exports attested artifacts, and cleans every resource. The design document should state remaining risks and compatibility tradeoffs. No architecture eliminates uncertainty; a good harness makes uncertainty bounded, observable, and governable.