Advanced project·3-4 hours

Reliable Agent Harness Project

Design a bounded tool-using agent harness with schemas, budgets, approvals, deterministic mocks, and failure traces.

Scenario

An agent may read tickets and draft updates but must never send or mutate external systems without approval. Dependencies can time out or return malformed data.

You will demonstrate

Specify tool permissions and schemas.
Enforce call and retry budgets.
Create reproducible failure tests and traces.

Project evidence

Complete each deliverable.

Deliverable 1

Define tool contracts

Specify validated inputs, outputs, errors, timeouts, and side-effect class.

Evidence: Two read tools and one approval-gated write tool.

Deliverable 2

Enforce budgets

Limit total calls and retries and return an explicit terminal state.

Evidence: Passing budget simulation tests.

Deliverable 3

Inject failures

Test timeout, malformed output, denied approval, and duplicate request.

Evidence: Deterministic traces for every failure.

Deliverable 4

Define a release gate

Set task success, unsafe-action, and budget-exhaustion thresholds.

Evidence: An automated go/no-go contract.

Runnable Python lab

Enforce a retry budget

Simulate tool outcomes and stop after success or a strict total-call budget.

Test output

Run the code when you are ready. The first load downloads the in-browser Python runtime.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

{'status': 'ok', 'calls': 3}
{'status': 'budget_exhausted', 'calls': 2}
ALL TESTS PASSED

Project defense

What is the safest default for a tool that sends an external message?

Assessment rubric

Every side effect has an explicit trust boundary.
Budgets terminate loops deterministically.
Mocks make failures reproducible.
Traces support root-cause diagnosis without secrets.

Useful references

OWASP Top 10 for LLM ApplicationsSecurity risks and mitigations for LLM and agent systems.

Project completion gate

Complete all deliverables, pass the automated code checks, and defend the key decision.

Deliverables 0/4Code pendingDefense pending