1
Define the event
Specify forecast horizon, eligible population, and when the outcome becomes known.
Required evidence: An event contract without ambiguous conditioning.
0/80 minimum characters
Builder mini-project·90-120 minutes
Uncertainty & Calibration Lab
Compare two probability forecasts using proper scoring, calibration groups, and a decision threshold.
Scenario
A service predicts whether an incident will occur in the next hour. Operations needs probabilities for staffing, not only a ranked alert list.
You will demonstrate
- Compute Brier score from first principles.
- Distinguish discrimination from calibration.
- Recommend a probability model with uncertainty caveats.
Project evidence
Show the work, not a checked box.
Each response is stored in this browser as you type. Include metrics, test output, or a decision rationale wherever the deliverable asks for it.
2
Score forecasts
Compute Brier score for both candidates and compare with the prevalence baseline.
Required evidence: A reproducible score table.
0/80 minimum characters
3
Inspect calibration
Group predictions into defensible bins and compare mean forecast with event rate.
Required evidence: A calibration table with sample sizes.
0/80 minimum characters
4
Propose an action policy
Select an alert threshold using staffing capacity and error costs.
Required evidence: Threshold, expected action volume, and fallback.
0/80 minimum characters
Runnable Python lab
Compare probabilistic forecasts
Implement Brier score and select the lower-loss candidate.
Project defense
Model A ranks every positive above every negative but predicts 0.99 for all positives and 0.80 for all negatives. What can still be wrong?
Rubric self-review
Rate the evidence, not your effort: 0 missing, 1 weak, 2 adequate, 3 strong. All criteria must be reviewed, but a low honest score does not get hidden.
The event and conditioning set are explicit.
Proper scoring is calculated correctly.
Small-bin uncertainty is acknowledged.
The policy separates probability quality from action choice.
Useful references
Project completion gate
Completion is controlled by stored evidence, deterministic tests, a decision defense, rubric review, and the artifact when required.
Evidence pendingCode pendingDefense pendingRubric pending
Optional cloud portfolio
Submit evidence across devices.
An account is required. Submit only when the local completion gate passes. AI review is advisory and separate from deterministic completion.