1
Build query slices
Include exact identifiers, paraphrases, broad topics, and unanswerable queries.
Required evidence: A versioned query set with relevance labels.
0/80 minimum characters
Advanced mini-project·90-120 minutes
Retrieval Quality System Project
Build an evaluation contract for lexical, dense, hybrid, and reranked retrieval over a small knowledge collection.
Scenario
A support search service misses exact product codes with dense search and paraphrases with keyword search. You must justify a hybrid design using query-level evidence.
You will demonstrate
- Implement Recall@k and reciprocal rank.
- Create query slices and relevance judgments.
- Separate retriever quality from answer quality.
Project evidence
Show the work, not a checked box.
Each response is stored in this browser as you type. Include metrics, test output, or a decision rationale wherever the deliverable asks for it.
2
Compute retrieval metrics
Calculate Recall@k and reciprocal rank per query and slice.
Required evidence: Passing metric tests and a score table.
0/80 minimum characters
3
Run an ablation
Compare lexical, dense, hybrid, and reranked results.
Required evidence: One controlled component comparison.
0/80 minimum characters
4
Propose production controls
Define index freshness, latency budget, fallback, and monitoring.
Required evidence: A retrieval architecture note.
0/80 minimum characters
Runnable Python lab
Implement Recall@k and reciprocal rank
Score ranked document IDs against a set of relevant documents.
Project defense
Why can a high average Recall@5 still hide a serious retrieval defect?
Rubric self-review
Rate the evidence, not your effort: 0 missing, 1 weak, 2 adequate, 3 strong. All criteria must be reviewed, but a low honest score does not get hidden.
Relevance judgments are explicit.
Metrics are computed per query before aggregation.
Ablations change one component at a time.
Freshness and failure behavior are included.
Useful references
Project completion gate
Completion is controlled by stored evidence, deterministic tests, a decision defense, rubric review, and the artifact when required.
Evidence pendingCode pendingDefense pendingRubric pending
Optional cloud portfolio
Submit evidence across devices.
An account is required. Submit only when the local completion gate passes. AI review is advisory and separate from deterministic completion.