Natural Language Processing · M2 · read

Perplexity: meaning, formula and four complete examples

Perplexity, log probabilities and smoothing

Unit 3, pages 121–156

Intrinsic evaluation evaluates a model directly on a held-out test corpus, for example using perplexity. Extrinsic evaluation places it inside a downstream application, such as a speech recognizer or translation system, and measures the application's performance. Extrinsic testing is closer to actual use but often slower and more expensive. A lower perplexity is not a guarantee of improvement on every downstream task.

Perplexity is the inverse probability of a test sequence, normalized by the number of prediction events. Intuitively, it is the model's effective average number of equally plausible choices. A model that always gives the correct next word probability 1/4 has perplexity 4. Lower is better when the models are tested on the same data, vocabulary, tokenization and boundary convention.

LaTeX: \operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}}

For calculator entry, use parentheses: P raised to (−1 ÷ N), or (1 ÷ P) raised to (1 ÷ N). Retain several digits in intermediate products. The ordinary start marker is usually given context, not a predicted event; the end marker counts if its probability is included.

Question. A model assigns probability 0.0625 to a test sequence containing four scored tokens. Compute its perplexity.

Given

P(W)=0.0625=1/16; N=4.

To find

PP(W).

Formula and why it applies

LaTeX: \operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N}

The root removes the effect of sequence length; the reciprocal makes lower surprise correspond to lower perplexity.

Solution

Step 1. Take the reciprocal

LaTeX: 1/P(W)=1/0.0625=16

Step 2. Use the fourth root

LaTeX: \operatorname{PP}=16^{1/4}

Step 3. Evaluate

LaTeX: 2^4=16\quad\Rightarrow\quad\operatorname{PP}=2

Final answer. Perplexity = 2.

Question. Two language models score the same three-token test sequence. Model A gives next-token probabilities 0.5, 0.5, 0.5. Model B gives 0.5, 0.25, 0.25. Compute their perplexities and select the better intrinsic result.

Given

Same sequence and N=3 for both models. A factors: 0.5,0.5,0.5. B factors: 0.5,0.25,0.25.

To find

Both sequence probabilities and perplexities.

Formula and why it applies

LaTeX: P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3}

A fair comparison uses identical test tokens and scoring conventions.

Solution

Step 1. Calculate A's probability

LaTeX: P_A=0.5\times0.5\times0.5=0.125=1/8

Step 2. Calculate A's perplexity

LaTeX: \operatorname{PP}_A=8^{1/3}=2

Step 3. Calculate B's probability

LaTeX: P_B=0.5\times0.25\times0.25=0.03125=1/32

Step 4. Calculate B's perplexity

LaTeX: \operatorname{PP}_B=32^{1/3}\approx3.174802

Step 5. Compare

2 is lower than 3.174802; A places more probability on the observed sequence.

Final answer. Model A is better by perplexity on this test: 2 versus about 3.174802.

Question. The bigram training corpus is “This is a dog”, “This is a cat”, “I love my cat”, “This is my name”. Score “This is my cat”, including an end marker, and compute perplexity using five predicted events.

Given

Each training sentence has one start and end marker. Test predictions: This, is, my, cat, </s>. Thus N=5.

To find

Bigram sentence probability and perplexity.

Formula and why it applies

LaTeX: P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5}

Start is given context. Four words plus the end marker produce five probability factors.

Solution

Step 1. Count the required transitions

Transition	Pair count / history count
<s>→This	3/4
This→is	3/3
is→my	1/3
my→cat	1/2
cat→</s>	2/2

Step 2. Form the product

LaTeX: P=\frac34\times1\times\frac13\times\frac12\times1=\frac18

Step 3. Take the reciprocal

LaTeX: 1/P=8

Step 4. Take the fifth root

LaTeX: \operatorname{PP}=8^{1/5}\approx1.515717

Final answer. P(W)=1/8 and perplexity≈1.515717 using five predicted events.

If the question explicitly states N = 6: use that supplied convention and report PP = 8^1/6 ≈ 1.414214. The class slide counts both boundary markers in N. Write the chosen N beside the formula; do not silently switch between 5 and 6. Also, This occurs three times and is follows it three times, so P(is|This)=3/3, not 3/4.

Question. A uniform model assigns probability 1/10 to each actual next word in a five-token test. Find its perplexity.

Given

Five scored tokens, each with assigned probability 0.1. N=5.

To find

The sentence probability and perplexity.

Formula and why it applies

LaTeX: P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5}

Uniform equal probabilities make the effective number of choices especially clear.

Solution

Step 1. Multiply the five factors

LaTeX: P(W)=0.1^5=0.00001=10^{-5}

Step 2. Invert

LaTeX: 1/P(W)=10^5

Step 3. Take the fifth root

LaTeX: \operatorname{PP}=(10^5)^{1/5}=10

Final answer. Perplexity = 10.

Yaad rakho: Probability high → surprise low → perplexity low.