Evaluate outcomes, trajectories, and effects

Agent evaluation must answer a harder question than “Was the final text good?” An agent operates over time, changes an environment, and encounters observations that depend on earlier actions. Evaluation therefore needs at least three layers. Outcome evaluation checks whether the requested state was achieved. Trajectory evaluation checks whether the route was legal, efficient, and recoverable. Effect evaluation checks what durable changes occurred, including unintended ones. A system can pass the first layer while failing the other two: it may solve a task after leaking data, spending an excessive budget, or creating and then deleting the wrong record.

Begin with an executable task contract. Pin the starting environment, fixtures, credentials, time assumptions, tool and policy versions, model configuration, and resource budgets. Define success predicates over observable state, not prose. Define safety invariants that must hold on every step and forbidden effects that fail the case even if the goal is reached. Define acceptable terminal reasons for underspecified or unauthorized requests. This converts a demonstration into a repeatable experiment.

SWE-bench illustrates executable outcome evaluation: a repository issue is paired with a codebase and tests so a proposed change can be checked. AgentBench emphasizes interactive environments and long-horizon behavior across domains. GAIA emphasizes questions whose answers require reasoning, tools, and real-world knowledge. These benchmarks measure different constructs; none is a universal score for “agency.” A customer-support system needs domain cases, policies, authentication states, and ledger effects. A coding system needs repository setup, reproducible dependencies, relevant tests, change-scope checks, and secret-exfiltration probes.

Build a case matrix across ordinary success, ambiguous goals, missing data, permission denial, tool unavailability, partial external failure, stale memory, adversarial content, budget pressure, and cancellation. Vary one factor at a time for diagnosis, then include realistic combinations for robustness. Include counterfactual pairs where a small but policy-relevant difference must change the action—for example, the same refund request one day inside and one day outside the allowed window. These pairs reveal whether the system uses evidence or merely follows a familiar narrative.

Use deterministic oracles whenever possible: tests, database queries, schema checks, policy engines, file diffs, checksums, event counts, or simulated ledgers. A model judge is useful for qualities without a complete mechanical oracle, such as whether a clarification is understandable, but it introduces its own variance and bias. Calibrate judges against human-labelled examples, blind them to irrelevant model identity, use structured rubrics, measure agreement, and keep safety-critical predicates deterministic. Never let the same free-form model claim both the work and the proof.

Score distributions, not one run. Agent policies and environments can be stochastic, so run repeated trials with controlled seeds where the stack supports them. Report verified success rate, invariant-violation rate, escalation correctness, median and tail latency, tool calls, model tokens, external writes, and cost. Pass@k answers whether at least one of k attempts succeeds; it can flatter systems that are unreliable on a single production attempt and may multiply harmful effects. For autonomous operation, pass@1, worst-case safety, and recovery quality usually matter more.

Trajectory assertions turn traces into test evidence. Examples include “authorization precedes every write,” “the same failed tool and equivalent arguments do not occur more than twice,” “no secret value appears in a model request,” and “completion follows a verifier event.” Avoid grading private chain-of-thought. Record operational events: normalized proposal, validation decision, authorized call, result class, reducer transition, verifier output, budget delta, and terminal reason. This is sufficient to test control behavior without requiring hidden reasoning.

Evaluation sets need governance. Separate development, regression, release, red-team, and shadow sets. Version cases and fixtures; record why each exists and which incident or requirement it covers. Prevent benchmark leakage by limiting access to held-out cases and by rotating realistic variants. When a production failure occurs, minimize it into a reproducible regression case while protecting personal data. Track score changes against exact model, prompt, tool, policy, retrieval-index, and sandbox versions so improvements are attributable.

A release gate should combine capability and risk. Require minimum task success by slice, zero critical invariant violations, bounded regression from the current production system, successful recovery tests, acceptable cost and latency, and sign-off from the named risk owner for residual limitations. A new model that raises average success but violates one payment invariant is not an upgrade. Use canary traffic or shadow execution with writes disabled before widening authority. Roll out tool and policy changes independently where possible so a regression has a small search space.

Operations extend evaluation into live service. Define service-level indicators such as verified completion rate, correct escalation rate, no-progress termination rate, ambiguous-effect backlog, p95 run duration, cost per verified success, policy-denial rate, and operator interventions. High tool success can coexist with poor task success; low cost can reflect premature stopping. Dashboards should preserve these relationships and support slices by task type, tenant risk tier, model, tool version, and terminal reason.

Alert on invariant violations, write-after-cancel attempts, repeated approval failures, unexplained authority denials, trace discontinuity, abnormal egress, and rising reconciliation queues. Sample normal traces for quality review, but retain all high-risk and anomalous traces according to privacy policy. Redact secrets at collection, restrict trace access, and make retention deliberate; observability data can contain the same sensitive context as prompts.

An incident loop closes the discipline. Contain capabilities, reconcile external state, communicate impact, preserve evidence, identify the failed assumption, update the threat model and runbook, add a regression case, and verify the remediation under fault injection. The mature measure of an agent is not that it never encounters uncertainty. It is that uncertainty becomes a bounded, visible state with evidence, ownership, and a safe next action.