1
Specify answer behavior
Define supported answer, partial answer, clarification, and abstention.
Required evidence: A response schema with examples.
0/80 minimum characters
Advanced mini-project·3-4 hours
Grounded AI Assistant Project
Design and evaluate a small assistant that answers only from supplied evidence, cites sources, and abstains when support is missing.
Scenario
An internal policy assistant must answer employee questions without inventing policy. Latency matters, but unsupported claims are the primary failure.
You will demonstrate
- Define claim-level grounding.
- Build answer, citation, and abstention tests.
- Design a bounded generation and fallback contract.
Project kit
Project evidence
Show the work, not a checked box.
Each response is stored in this browser as you type. Include metrics, test output, or a decision rationale wherever the deliverable asks for it.
2
Create a golden set
Write answerable, ambiguous, unanswerable, and adversarial questions.
Required evidence: At least four cases with evidence and expected behavior.
0/80 minimum characters
3
Measure grounding
Compute citation coverage and unsupported-claim rate.
Required evidence: Per-case traces and aggregate metrics.
0/80 minimum characters
4
Set a release gate
Choose quality thresholds and a rollback response.
Required evidence: A gate that can fail automatically.
0/80 minimum characters
Runnable Python lab
Compute citation coverage
Measure the fraction of answer claims linked to an allowed evidence identifier.
Project defense
An answer is fluent and correct according to general knowledge but unsupported by the supplied policy passages. How should a grounded-system evaluator score it?
Rubric self-review
Rate the evidence, not your effort: 0 missing, 1 weak, 2 adequate, 3 strong. All criteria must be reviewed, but a low honest score does not get hidden.
Claims map to evidence spans.
Unanswerable cases are first-class tests.
Metrics distinguish retrieval and generation failures.
The design includes abstention and rollback.
Artifact
Attach the portfolio evidence.
Submit a notebook, Markdown report, or PDF containing versioned traces and aggregate results. Do not include credentials or private user data.
Anonymous mode stores only metadata and a fingerprint locally. Private upload requires an account and an explicit file selection. Uploaded notebooks are never executed.
Useful references
Project completion gate
Completion is controlled by stored evidence, deterministic tests, a decision defense, rubric review, and the artifact when required.
Evidence pendingCode pendingDefense pendingRubric pendingArtifact pending
Optional cloud portfolio
Submit evidence across devices.
An account is required. Submit only when the local completion gate passes. AI review is advisory and separate from deterministic completion.