Builder project·2-3 hours

Classical Model Selection Project

Compare a linear rule, a decision stump, and a nearest-centroid classifier under one shared evaluation contract.

Scenario

A small operations team needs a transparent binary classifier and cannot support a large serving stack. Your recommendation must balance error cost, stability, and explainability.

You will demonstrate

Use identical splits and metrics for every candidate.
Diagnose model-specific failure modes.
Choose complexity only when evidence justifies it.

Project evidence

Complete each deliverable.

Deliverable 1

Lock the baseline

Define a deterministic or majority baseline before candidate comparison.

Evidence: Baseline predictions and confusion counts.

Deliverable 2

Compare candidates

Evaluate all candidates on identical held-out rows and threshold policy.

Evidence: One scorecard with error counts and cost.

Deliverable 3

Stress the decision

Perturb one feature and identify which model changes predictions unexpectedly.

Evidence: A small sensitivity table.

Deliverable 4

Defend the selection

Recommend a model or the baseline and name a rollback trigger.

Evidence: Decision memo tied to the scorecard.

Runnable Python lab

Select by cost, then simplicity

Compute cost from fixed confusion counts and break a tie in favor of fewer fitted parameters.

Test output

Run the code when you are ready. The first load downloads the in-browser Python runtime.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

[('linear', 16), ('tree', 16), ('centroid', 17)]
ALL TESTS PASSED

Project defense

Why must every candidate use the same held-out rows?

Assessment rubric

Comparisons use identical evidence.
Complexity is treated as a cost.
Failure behavior is demonstrated, not merely named.
The recommendation is reversible.

Useful references

Model Cards for Model ReportingA reporting framework for intended use, evaluation context, limitations, and ethical considerations.

Project completion gate

Complete all deliverables, pass the automated code checks, and defend the key decision.

Deliverables 0/4Code pendingDefense pending