Builder project·90-120 minutes

Uncertainty & Calibration Lab

Compare two probability forecasts using proper scoring, calibration groups, and a decision threshold.

Scenario

A service predicts whether an incident will occur in the next hour. Operations needs probabilities for staffing, not only a ranked alert list.

You will demonstrate

Project evidence

Complete each deliverable.

Deliverable 1

Specify forecast horizon, eligible population, and when the outcome becomes known.

Evidence: An event contract without ambiguous conditioning.

Deliverable 2

Compute Brier score for both candidates and compare with the prevalence baseline.

Evidence: A reproducible score table.

Deliverable 3

Group predictions into defensible bins and compare mean forecast with event rate.

Evidence: A calibration table with sample sizes.

Deliverable 4

Select an alert threshold using staffing capacity and error costs.

Evidence: Threshold, expected action volume, and fallback.

Runnable Python lab

Implement Brier score and select the lower-loss candidate.

Test output

Run the code when you are ready. The first load downloads the in-browser Python runtime.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

B has the lower Brier score.
ALL TESTS PASSED

Project defense

Complete all deliverables, pass the automated code checks, and defend the key decision.

Deliverables 0/4Code pendingDefense pending