Trajectory evaluation, counterfactual replay, and red teams

Agent evaluation must judge both outcome and trajectory. A task can succeed through an unauthorized or wasteful path; it can fail safely after correctly refusing an impossible request; it can produce the right prose while never performing the claimed effect. Define success from externally verifiable state and evaluate the sequence of evidence, decisions, actions, costs, and policy outcomes that led there.

Build task fixtures with initial state, goal contract, authorized identity, tool versions, graph and ontology releases, budgets, injected events, expected terminal classes, required evidence, forbidden actions, and acceptable alternative plans. Use recorded or simulated tool observations for repeatability. Maintain live end-to-end tests separately to detect adapter and environment drift.

Outcome metrics include verified task completion, answer correctness and support, safe abstention, and effect accuracy. Trajectory metrics include valid proposal rate, evidence sufficiency before action, unnecessary calls, tool and model cost, latency, replans, repeated states, approval compliance, irreversible actions, and recovery after failure. Security metrics count unauthorized proposals and effects separately: a denied bad proposal is a model-quality issue but a successful bad effect is a control failure.

Evaluate by slices: task length, write risk, ambiguity, graph hop count, conflicting evidence, stale state, rare entity, permission boundary, tool outage, and adversarial content. AgentBench illustrates evaluation across interactive environments, but a product suite must reflect its own tools, policies, and harm. Average completion can hide catastrophic behavior on a small high-risk slice.

Counterfactual replay changes one component while holding recorded environment responses fixed. Compare a new planner, prompt, model, policy, or ontology against the same trajectory observations. This identifies whether it would choose a safer or more efficient action without repeating effects. Replay has limits: a different early action would have produced a different environment, so branch simulations need causal models or fresh sandbox runs. Label off-policy conclusions accordingly.

Perturbation tests vary entity labels, source order, irrelevant text, tool response order, latency, and equivalent schemas. Stable systems should preserve material decisions. Metamorphic relations offer useful oracles: removing authorized evidence should not increase confidence; adding a duplicate source should not count as independent support; revoking a permission should never add a capability; replacing a label while preserving IRI should not change target identity.

Red-team trajectories chain weaknesses. Insert an indirect injection into an authorized document, make it propose a recipient expansion, cause a tool timeout, and test whether a retry duplicates the send. Poison a graph edge, then observe its effect on planning and approval. Attempt to store the attack in memory. Each attack names the expected stopping control and asserts zero unauthorized effects.

Human review uses structured rubrics and evidence traces. Reviewers see goal, authorized scope, relevant observations, proposals, policy decisions, receipts, and terminal verification—not hidden chain-of-thought. Measure agreement and adjudicate disputed labels. Protect test fixtures and canaries from training or prompt leakage.

Online monitoring complements evaluation with task success confirmations, reversals, user corrections, approval denials, incident reports, cost, latency, and terminal reasons. Canary new models or policies on read-only or shadow traffic before expanding effect permissions. Automatic rollback triggers include supported-answer regression, increased ambiguous writes, policy-control failure, or budget explosion.

The evaluation report should state releases, environment, judge versions, confidence intervals, excluded cases, known limitations, and failures. Never claim “safe” from a finite suite. The goal is evidence that defined controls work across known and newly discovered failure classes, with a process that turns every incident into a regression.