Natural Language Processing · M2 · formulas

Perplexity: meaning, formula and four complete examples

Smoothing & perplexity

Formulas explained through full worked examples

Intrinsic evaluation evaluates a model directly on a held-out test corpus, for example using perplexity. Extrinsic evaluation places it inside a downstream application, such as a speech recognizer or translation system, and measures the application's performance. Extrinsic testing is closer to actual use but often slower and more expensive. A lower perplexity is not a guarantee of improvement on every downstream task.

Perplexity is the inverse probability of a test sequence, normalized by the number of prediction events. Intuitively, it is the model's effective average number of equally plausible choices. A model that always gives the correct next word probability 1/4 has perplexity 4. Lower is better when the models are tested on the same data, vocabulary, tokenization and boundary convention.

\operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}}LaTeX: \operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}}
SymbolMeaning
WThe complete test sequence.
P(W)Product of the probabilities assigned to the actual next tokens.
NNumber of scored prediction events under the stated convention.
Exponent −1/NTake the reciprocal, then the Nth root.

For calculator entry, use parentheses: P raised to (−1 ÷ N), or (1 ÷ P) raised to (1 ÷ N). Retain several digits in intermediate products. The ordinary start marker is usually given context, not a predicted event; the end marker counts if its probability is included.

Question. A model assigns probability 0.0625 to a test sequence containing four scored tokens. Compute its perplexity.

Given

P(W)=0.0625=1/16; N=4.

To find

PP(W).

Formula and why it applies

\operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N}LaTeX: \operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N}

The root removes the effect of sequence length; the reciprocal makes lower surprise correspond to lower perplexity.

Solution

Step 1. Take the reciprocal
1/P(W)=1/0.0625=16LaTeX: 1/P(W)=1/0.0625=16
Step 2. Use the fourth root
\operatorname{PP}=16^{1/4}LaTeX: \operatorname{PP}=16^{1/4}
Step 3. Evaluate
2^4=16\quad\Rightarrow\quad\operatorname{PP}=2LaTeX: 2^4=16\quad\Rightarrow\quad\operatorname{PP}=2
Final answer. Perplexity = 2.

Question. Two language models score the same three-token test sequence. Model A gives next-token probabilities 0.5, 0.5, 0.5. Model B gives 0.5, 0.25, 0.25. Compute their perplexities and select the better intrinsic result.

Given

Same sequence and N=3 for both models. A factors: 0.5,0.5,0.5. B factors: 0.5,0.25,0.25.

To find

Both sequence probabilities and perplexities.

Formula and why it applies

P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3}LaTeX: P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3}

A fair comparison uses identical test tokens and scoring conventions.

Solution

Step 1. Calculate A's probability
P_A=0.5\times0.5\times0.5=0.125=1/8LaTeX: P_A=0.5\times0.5\times0.5=0.125=1/8
Step 2. Calculate A's perplexity
\operatorname{PP}_A=8^{1/3}=2LaTeX: \operatorname{PP}_A=8^{1/3}=2
Step 3. Calculate B's probability
P_B=0.5\times0.25\times0.25=0.03125=1/32LaTeX: P_B=0.5\times0.25\times0.25=0.03125=1/32
Step 4. Calculate B's perplexity
\operatorname{PP}_B=32^{1/3}\approx3.174802LaTeX: \operatorname{PP}_B=32^{1/3}\approx3.174802
Step 5. Compare

2 is lower than 3.174802; A places more probability on the observed sequence.

Final answer. Model A is better by perplexity on this test: 2 versus about 3.174802.

Question. The bigram training corpus is “This is a dog”, “This is a cat”, “I love my cat”, “This is my name”. Score “This is my cat”, including an end marker, and compute perplexity using five predicted events.

Given

Each training sentence has one start and end marker. Test predictions: This, is, my, cat, </s>. Thus N=5.

To find

Bigram sentence probability and perplexity.

Formula and why it applies

P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5}LaTeX: P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5}

Start is given context. Four words plus the end marker produce five probability factors.

Solution

Step 1. Count the required transitions
TransitionPair count / history count
<s>→This3/4
This→is3/3
is→my1/3
my→cat1/2
cat→</s>2/2
Step 2. Form the product
P=\frac34\times1\times\frac13\times\frac12\times1=\frac18LaTeX: P=\frac34\times1\times\frac13\times\frac12\times1=\frac18
Step 3. Take the reciprocal
1/P=8LaTeX: 1/P=8
Step 4. Take the fifth root
\operatorname{PP}=8^{1/5}\approx1.515717LaTeX: \operatorname{PP}=8^{1/5}\approx1.515717
Final answer. P(W)=1/8 and perplexity≈1.515717 using five predicted events.

If the question explicitly states N = 6: use that supplied convention and report PP = 81/6 ≈ 1.414214. The class slide counts both boundary markers in N. Write the chosen N beside the formula; do not silently switch between 5 and 6. Also, This occurs three times and is follows it three times, so P(is|This)=3/3, not 3/4.

Question. A uniform model assigns probability 1/10 to each actual next word in a five-token test. Find its perplexity.

Given

Five scored tokens, each with assigned probability 0.1. N=5.

To find

The sentence probability and perplexity.

Formula and why it applies

P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5}LaTeX: P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5}

Uniform equal probabilities make the effective number of choices especially clear.

Solution

Step 1. Multiply the five factors
P(W)=0.1^5=0.00001=10^{-5}LaTeX: P(W)=0.1^5=0.00001=10^{-5}
Step 2. Invert
1/P(W)=10^5LaTeX: 1/P(W)=10^5
Step 3. Take the fifth root
\operatorname{PP}=(10^5)^{1/5}=10LaTeX: \operatorname{PP}=(10^5)^{1/5}=10
Final answer. Perplexity = 10.

Yaad rakho: Probability high → surprise low → perplexity low.