Tool orchestration, budgets, and durable execution

Tools are capability boundaries. Each tool adapter exposes a small typed operation, validates arguments, enforces identity and scope, invokes an external system, and returns a typed observation. It should hide dangerous implementation flexibility from the model. A database tool offers approved queries or parameterized operations, not unrestricted SQL; a file tool receives a validated workspace-relative target, not an arbitrary shell command.

The registry defines operation name and version, input and output schemas, side-effect class, required scopes, timeout, retry behavior, rate and cost limits, idempotency support, sandbox, and evidence fields. The model sees only tools allowed for the task and current state. Tool descriptions are documentation, not enforcement; the runtime validates every proposal even if the description said “read only.”

Normalize observations. Distinguish success, no-result, invalid-request, denied, retryable failure, permanent failure, ambiguous outcome, and partial result. Include request identifier, external version or transaction, time, pagination, truncation, and provenance. Do not convert all exceptions into prose. The controller needs typed categories to select a safe transition.

Durable execution persists state before and after external effects. A workflow system such as Temporal records event history and replays deterministic workflow logic while activities perform nondeterministic external work. The general pattern applies without a specific framework: persist intent, execute with an idempotency key, persist observation, then advance state. If the worker crashes, the controller can determine whether to retry, observe, compensate, or escalate.

Retries depend on semantics. Read operations may retry with exponential backoff and jitter within freshness bounds. An idempotent write can retry under the same key. A non-idempotent or ambiguously completed write must be observed before any repeat. Permanent validation and authorization failures do not improve with retry. Limit attempts and elapsed time; route persistent failures to a terminal or human-review state.

Budgets are multidimensional and hierarchical. A task budget allocates model tokens, tool calls, elapsed time, monetary cost, retrieved evidence, and side effects. Each plan step reserves an estimate; execution records actual use; replanning sees the remainder. Per-tenant and global quotas protect shared systems. High-risk actions may have a count of zero until approval grants a narrowly scoped capability.

Concurrency is useful for independent reads but unsafe when actions share mutable state. The plan encodes dependencies and the controller attaches state versions. Parallel observations merge only when their scopes are compatible. Writes to one resource use serialization, etags, or compare-and-set. Cancellation propagates to queued activities where possible and prevents newly returned results from authorizing a canceled task.

Sandboxes limit code, browser, and data-processing tools. Restrict filesystem roots, network destinations, credentials, CPU, memory, wall time, and output size. Start with no ambient secrets. Provide short-lived credentials scoped to the exact operation. Treat tool output as untrusted even when execution is sandboxed; it can still contain malicious instructions or sensitive data.

Observability follows the transition: task, state version, plan step, proposal, policy decision, tool request, attempt, observation, evidence update, budget delta, and next state. Use OpenTelemetry-compatible traces where practical while redacting content. Metrics include success by terminal reason, action denial, retries, ambiguous outcomes, idempotency collisions, budget exhaustion, latency, cost, and no-progress stops.

Test adapters with contract fixtures, timeouts, malformed results, pagination, partial failures, permission denial, duplicated responses, and crash recovery. Then test the workflow with recorded observations so model and controller changes can be replayed without repeating effects. Reliable orchestration is less about how many tools an agent can call and more about how precisely the system controls one call when conditions are uncertain.