Audit two candidate classifiers and defend a deployment threshold using evidence rather than one headline score.
For a small example, two classifiers trade recall for precision on 20 examples. Build a confusion matrix at both thresholds and state which mistake matters more for the decision.
A candidate model appears 4% better than baseline. Reproduce the split, verify leakage controls, add confidence intervals, segment errors, and document the chosen operating point.
Initializing Interactive Playground...
Predict how one control changes the Model Evaluation Project result.
The starter deliberately follows a tempting but flawed assumption: Picking the model with the largest accuracy
Check: The browser checks pass and you can name the failure mechanism before opening the explanation.
Failing starter lab
Compute the lowest-cost feasible candidate-threshold pair under an explicit false-negative cost and review-capacity constraint.
Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.
Explanations after each answer · 80% to pass · Model Evaluation Project
Chapter 1 of 2 · Core
You are reviewing two fixed classifiers for a limited support-triage launch. Missing a truly urgent case costs five times as much as sending a normal case for manual review. Review capacity allows at most four positive predictions in this eight-case sample.
Your work must be reproducible. Compute confusion counts at candidate thresholds, report precision and recall, check the high-priority subgroup separately, and recommend launch, revise, or stop. Do not select by accuracy alone.
Ideas to understand
ROC and AUC, Clearly Explained!
StatQuest
Interpret threshold tradeoffs and understand what ROC-AUC does and does not establish about a model.
Primary sources and free technical reading for Model Evaluation Project.