Advanced project·3-4 hours

Retrieval Quality System Project

Build an evaluation contract for lexical, dense, hybrid, and reranked retrieval over a small knowledge collection.

Scenario

A support search service misses exact product codes with dense search and paraphrases with keyword search. You must justify a hybrid design using query-level evidence.

You will demonstrate

Implement Recall@k and reciprocal rank.
Create query slices and relevance judgments.
Separate retriever quality from answer quality.

Project evidence

Complete each deliverable.

Deliverable 1

Build query slices

Include exact identifiers, paraphrases, broad topics, and unanswerable queries.

Evidence: A versioned query set with relevance labels.

Deliverable 2

Compute retrieval metrics

Calculate Recall@k and reciprocal rank per query and slice.

Evidence: Passing metric tests and a score table.

Deliverable 3

Run an ablation

Compare lexical, dense, hybrid, and reranked results.

Evidence: One controlled component comparison.

Deliverable 4

Propose production controls

Define index freshness, latency budget, fallback, and monitoring.

Evidence: A retrieval architecture note.

Runnable Python lab

Implement Recall@k and reciprocal rank

Score ranked document IDs against a set of relevant documents.

Test output

Run the code when you are ready. The first load downloads the in-browser Python runtime.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

recall@3: 0.5
rr: 0.5
ALL TESTS PASSED

Project defense

Why can a high average Recall@5 still hide a serious retrieval defect?

Assessment rubric

Relevance judgments are explicit.
Metrics are computed per query before aggregation.
Ablations change one component at a time.
Freshness and failure behavior are included.

Useful references

BEIRA heterogeneous benchmark and paper for information retrieval evaluation.

Project completion gate

Complete all deliverables, pass the automated code checks, and defend the key decision.

Deliverables 0/4Code pendingDefense pending