Probability

A sample space lists possible outcomes; an event is a set of those outcomes. For a fair six-sided die, the event “even” contains 2, 4 and 6, so its probability is 3/6. Counting works this way because the six outcomes are equally likely. For unequal outcomes, add their assigned probability mass instead.

Conditioning changes the denominator. Among 100 messages, 20 are spam; 16 spam messages and 8 other messages contain “offer”. Among the 24 offer messages, 16 are spam: P(spam | offer)=16/24=2/3. But P(offer | spam)=16/20=0.8. Reversing the condition answers a different question.

Disjoint events cannot occur together. Independent events satisfy P(A and B)=P(A)P(B). Two disjoint events with positive probability are not independent: observing one rules out the other.

Suppose 10 of 1,000 items have a defect. A test flags 9 of those 10 and also flags 99 of the 990 sound items. There are 108 positive tests, but only 9 correspond to defects. P(defect | positive)=9/108=1/12, about 8.3%, not 90%.

The sensitivity 9/10 is P(positive | defect); it is not the posterior. Bayes combines this likelihood with the 1% prior prevalence and both ways to get a positive test. If prevalence changes, the posterior can change even with the same sensitivity and false-positive rate. This arithmetic depends on the declared population and test behavior, not a universal property of all tests.

A language model assigns probability to each possible next token. To score an observed sequence, look up the probability assigned to each actual target—not just the largest probability in the row. Average the negative natural logarithms over all scored targets.

For observed probabilities 0.5 and 0.25, mean loss is (-ln 0.5-ln 0.25)/2=1.039721 nats per target. Perplexity is exp(loss)=2.828427. Averaging probabilities first gives a different, incorrect scoring rule. A zero probability for an observed target has infinite log loss; smoothing can avoid zero estimates in a count model.

Use the same tokenization, target weighting and document-boundary rules when comparing models. Lower held-out loss means better prediction on that evidence; it does not establish factual truth, useful reasoning or improvement on every population. Stage 00 puts these calculations into an actual uniform/unigram/bigram baseline.