Hypothesis Testing
p-values, Z-tests, and T-tests.
For a small example, a supposedly fair coin gives 9 heads in 10 flips. Define the null, compute a tail probability, and distinguish a low p-value from the probability the coin is fair.
Hypothesis Testing
In practice
A new onboarding flow raises conversion in a randomized test. Predefine outcome and stopping rule, estimate effect and interval, check power, and discuss practical significance.
Initializing Interactive Playground...
Experiment notes
Record your experiment
Predict how one control changes the Hypothesis Testing result.
Implementation
Repair a failing Hypothesis Testing starter
The starter deliberately follows a tempting but flawed assumption: Reading a p-value as the probability the null is true
- Write a prediction that makes the flawed assumption visible and names what should change.
- Manipulate one playground control while holding the others fixed.
- Save the result and explain why this correction works: Interpret it conditional on the null and pair it with effect uncertainty.
Check: Your saved hypothesis predicts a direction, the experiment tests it, and the result changes or supports your explanation.
Knowledge check
10 questions
Explanations after each answer · 80% to pass · Hypothesis Testing
Chapter 1 of 7 · Core
The Neyman-Pearson Framework and Statistical Decision Theory
Statistical hypothesis testing is formally structured as a decision-theoretic problem. Let represent a random sample drawn from a probability space where the distribution is indexed by a parameter vector . The parameter space is partitioned into two disjoint subsets: the null parameter space and the alternative parameter space , such that and . We define the null hypothesis and the alternative hypothesis .
A non-randomized statistical decision rule (or test) is a mapping , where denotes the rejection of (acceptance of ) and denotes a failure to reject . The critical region is the set of outcomes for which the null hypothesis is rejected: .
The performance of any test is fully characterized by its power function .
- For , represents the Type I Error Rate (false positive probability), which is the probability of rejecting a true null hypothesis. The significance level (or size) of the test is defined as the supremum of this error rate: .
- For , represents the Power of the test, while is the Type II Error Rate (false negative probability), which is the probability of failing to reject a false null hypothesis.
The Neyman-Pearson Lemma provides the fundamental theoretical justification for constructing optimal tests. For simple hypotheses versus , the most powerful test of size rejects if the likelihood ratio falls below a threshold : where is chosen such that .
For composite hypotheses, Uniformly Most Powerful (UMP) tests are generally unavailable unless the family of distributions possesses a Monotone Likelihood Ratio (MLR). In the general case, we construct tests using the Likelihood Ratio Test (LRT) statistic: Wilks' Theorem provides the asymptotic distribution under regular conditions. As , the test statistic converges in distribution to a chi-squared distribution under : where the degrees of freedom represents the number of constraints imposed by the null hypothesis.
Key equations
Video explanation
Hypothesis Testing and the Null Hypothesis, Clearly Explained!
StatQuest
Understand null hypotheses, evidence, and why a test result is not the same as practical importance.
References
Primary sources and free technical reading for Hypothesis Testing.