Hypothesis Testing

p-values, Z-tests, and T-tests.

Not started3 min explanation

Visualize, practice, and deep-dive material are optional—use only what helps you learn.

Explanation

A focused 3-minute explanation using the topic's authored material.

Learning goals and prerequisites

After this lesson

Formulate valid tests and interpret uncertainty
Design adequately powered experiments
Avoid common multiple-testing and peeking failures

Helpful before starting

Probability distributions and sampling
Expectation, variance, and standard error
Basic experimental design

Start here

Hypothesis Testing, in plain language

p-values, Z-tests, and T-tests. Hypothesis tests provide a disciplined way to distinguish signal from sampling noise and to govern experiments, launches, and model comparisons.

For a small example, a supposedly fair coin gives 9 heads in 10 flips. Define the null, compute a tail probability, and distinguish a low p-value from the probability the coin is fair. This is the mechanism to keep in view as the lesson becomes more technical. Before moving on, identify the input, transformation, output, and one observation that would falsify your conclusion.

Key points

Null and alternative hypotheses, test statistics, p-values, and confidence intervals.
Type I and II errors, power, effect size, and practical significance.
Z, t, chi-square, proportion, and nonparametric tests.

Use it well

When Hypothesis Testing helps—and where it breaks

A new onboarding flow raises conversion in a randomized test. Predefine outcome and stopping rule, estimate effect and interval, check power, and discuss practical significance. A useful result still depends on checking the assumptions and evidence below rather than treating one successful output as proof.

Key points

Reading a p-value as the probability the null is true. Better approach: Interpret it conditional on the null and pair it with effect uncertainty.
Stopping when significance appears. Better approach: Use a fixed horizon or valid sequential procedure.
Equating statistical with practical significance. Better approach: Tie minimum detectable effects to product value and risk.

Was this lesson helpful?

Submit to the team when server feedback is available; otherwise this browser keeps a local copy and tells you so.

Back to Topic Library