Reading tools and contents
4 min
Harness Engineering & Sandboxes

Chapter 9 of 10

Evaluation hooks and evidence-based verdicts

Evaluation

About 4 minutes · includes examples, an exercise, and references

Chapter at a glance

  • •Critical invariants gate verdicts before aggregate quality scores.
  • •Evaluator identity and raw evidence must be versioned and retained.
  • •Use paired, sliced, multi-metric comparisons for stochastic systems.

Evaluation hooks turn a runner into a learning and release system. A hook is a versioned observation point where the harness exposes normalized inputs, artifacts, and trace evidence to an evaluator without granting the evaluator authority to alter the run. Hooks can operate before execution, after each transition, at terminal proposal, after artifact export, or asynchronously on sampled production traces.

Separate invariants, task metrics, and policy judgments. Invariants are deterministic requirements such as schema validity, no forbidden path changes, all citations resolve, or resource budgets remain nonnegative. Task metrics measure quality such as test pass rate, retrieval recall, answer support, latency, or cost. Policy judgments decide whether evidence is sufficient for release, escalation, or human review. An average score must not override a failed critical invariant.

The evaluator contract names fixture version, evaluator code and model versions, metric definitions, thresholds, and required evidence. It returns structured metric values, confidence or uncertainty where available, slice labels, diagnostics, and a verdict. Store raw evaluator artifacts so a future metric implementation can be audited. If a model judge is used, preserve its prompt, model identifier, sampling settings, response, and parser result; pair it with deterministic checks and periodic human calibration.

Evaluation should be paired across candidate and baseline when possible. Run both on the same fixtures and external-response recordings. Compare outcome, cost, latency, safety, and recovery evidence by slice. Randomly varying fixtures or judges can hide regressions. Report confidence intervals or repeated-run distributions for stochastic systems rather than a single decimal score.

Hooks inside the execution loop can enforce gates. A transition evaluator may detect no progress, policy risk, or invalid evidence and prevent the next effect. Keep this evaluator bounded and deterministic where possible because it sits on the critical path. Rich semantic review can run after a checkpoint or terminal proposal. Define what happens when an evaluator is unavailable: fail closed, pause, use a preapproved degraded path, or continue only for low-consequence tasks. Never silently treat evaluation error as pass.

Evaluation datasets require provenance and coverage. Each case should identify source, consent or licensing constraints, task class, expected invariants, and known limitations. Include happy paths, rare but critical slices, adversarial inputs, dependency failures, cancellations, and ambiguous effects. HELM’s multi-metric and scenario-based framing is useful: no single benchmark captures capability, robustness, safety, fairness, and efficiency.

Avoid test leakage. Do not expose hidden evaluator fixtures or exact expected outputs to the planner unless the lesson intentionally teaches from them. For code tasks, run public tests during work and hidden tests in the evaluator environment. For security tasks, rotate adversarial cases and test generalized invariants. Keep evaluation credentials and datasets outside the task sandbox.

Production evaluation combines offline gates with online evidence. Shadow a candidate on recorded or mirrored tasks, canary a bounded tenant slice, and compare operational metrics. Sample complete traces for human review based on consequence and anomaly, not only random selection. Feedback should link to the exact run, artifacts, and evaluator version. A complaint without run identity is hard to learn from; a trace without user outcome is incomplete.

Evaluation hooks also enable fault-injection scoring. A resilience case can assert no duplicate writes, cleanup within a deadline, and complete evidence after a worker crash. This expands “quality” beyond final answer correctness. A model that achieves the artifact but routinely triggers retries, policy denials, or manual cleanup may be unsuitable for production.

Treat thresholds as release policy with ownership. Record why a threshold exists, which slices are critical, and who can approve an exception. Monitor metric drift and evaluator disagreement. When the evaluator changes, run it on a stable calibration set and compare old and new verdicts before replacing it. Versioning prevents historical reports from silently changing meaning.

Key points

  • Critical invariants gate verdicts before aggregate quality scores.
  • Evaluator identity and raw evidence must be versioned and retained.
  • Use paired, sliced, multi-metric comparisons for stochastic systems.
  • Evaluation errors require an explicit fail, pause, or degraded-mode policy.

Combine invariant and metric verdicts

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

Combine invariant and metric verdictspython
from dataclasses import dataclass
from math import isfinite

@dataclass(frozen=True)
class Evaluation:
    invariants: dict[str, bool]
    metrics: dict[str, float]

def verdict(result: Evaluation) -> str:
    if not all(result.invariants.values()):
        return "fail_invariant"
    task_success = result.metrics.get("task_success", float("nan"))
    cost_usd = result.metrics.get("cost_usd", float("nan"))
    if not isfinite(task_success) or not 0.0 <= task_success <= 1.0:
        return "fail_evaluator"
    if not isfinite(cost_usd) or cost_usd < 0.0:
        return "fail_evaluator"
    if task_success < 0.90:
        return "fail_quality"
    if cost_usd > 0.25:
        return "fail_budget"
    return "pass"

sample = Evaluation({"no_forbidden_writes": True}, {"task_success": .94, "cost_usd": .12})
print(verdict(sample))

Exercise

Specify an evaluator hook

Design the terminal evaluator for a repository-repair agent.

  1. Define critical invariants, task metrics, operational metrics, and slices.
  2. Specify hidden fixture isolation and evaluator failure behavior.
  3. Write a paired baseline comparison and exception policy.

Success criteria

  • A failed critical invariant cannot be averaged away.
  • The evaluator can be replayed from retained evidence.
  • Stochastic metrics report repeated-run uncertainty.

Reflect: Which current success metric rewards an outcome while ignoring unsafe or expensive trajectories?

References and further reading