Advanced project·3-4 hours

Grounded AI Assistant Project

Design and evaluate a small assistant that answers only from supplied evidence, cites sources, and abstains when support is missing.

Scenario

An internal policy assistant must answer employee questions without inventing policy. Latency matters, but unsupported claims are the primary failure.

You will demonstrate

Define claim-level grounding.
Build answer, citation, and abstention tests.
Design a bounded generation and fallback contract.

Project evidence

Complete each deliverable.

Deliverable 1

Specify answer behavior

Define supported answer, partial answer, clarification, and abstention.

Evidence: A response schema with examples.

Deliverable 2

Create a golden set

Write answerable, ambiguous, unanswerable, and adversarial questions.

Evidence: At least four cases with evidence and expected behavior.

Deliverable 3

Measure grounding

Compute citation coverage and unsupported-claim rate.

Evidence: Per-case traces and aggregate metrics.

Deliverable 4

Set a release gate

Choose quality thresholds and a rollback response.

Evidence: A gate that can fail automatically.

Runnable Python lab

Compute citation coverage

Measure the fraction of answer claims linked to an allowed evidence identifier.

Test output

Run the code when you are ready. The first load downloads the in-browser Python runtime.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

coverage: 0.667
ALL TESTS PASSED

Project defense

An answer is fluent and correct according to general knowledge but unsupported by the supplied policy passages. How should a grounded-system evaluator score it?

Assessment rubric

Claims map to evidence spans.
Unanswerable cases are first-class tests.
Metrics distinguish retrieval and generation failures.
The design includes abstention and rollback.

Useful references

Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksThe original RAG paper and evaluation framing.Model Cards for Model ReportingA reporting framework for intended use, evaluation context, limitations, and ethical considerations.

Project completion gate

Complete all deliverables, pass the automated code checks, and defend the key decision.

Deliverables 0/4Code pendingDefense pending