# Perplexity: meaning, formula and four complete examples Subject: Natural Language Processing Exam: TEE Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-smoothing/evaluation-perplexity Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-smoothing--evaluation-perplexity/ Formulas explained through full worked examples **Intrinsic evaluation** evaluates a model directly on a held-out test corpus, for example using perplexity. **Extrinsic evaluation** places it inside a downstream application, such as a speech recognizer or translation system, and measures the application's performance. Extrinsic testing is closer to actual use but often slower and more expensive. A lower perplexity is not a guarantee of improvement on every downstream task. **Perplexity** is the inverse probability of a test sequence, normalized by the number of prediction events. Intuitively, it is the model's effective average number of equally plausible choices. A model that always gives the correct next word probability 1/4 has perplexity 4. Lower is better when the models are tested on the same data, vocabulary, tokenization and boundary convention. $$ \operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}} $$ | Symbol | Meaning | | --- | --- | | W | The complete test sequence. | | P(W) | Product of the probabilities assigned to the actual next tokens. | | N | Number of scored prediction events under the stated convention. | | Exponent −1/N | Take the reciprocal, then the Nth root. | For calculator entry, use parentheses: P raised to (−1 ÷ N), or (1 ÷ P) raised to (1 ÷ N). Retain several digits in intermediate products. The ordinary start marker is usually given context, not a predicted event; the end marker counts if its probability is included. **Question. A model assigns probability 0.0625 to a test sequence containing four scored tokens. Compute its perplexity.** #### Given P(W)=0.0625=1/16; N=4. #### To find PP(W). #### Formula and why it applies $$ \operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N} $$ The root removes the effect of sequence length; the reciprocal makes lower surprise correspond to lower perplexity. #### Solution ##### Step 1. Take the reciprocal $$ 1/P(W)=1/0.0625=16 $$ ##### Step 2. Use the fourth root $$ \operatorname{PP}=16^{1/4} $$ ##### Step 3. Evaluate $$ 2^4=16\quad\Rightarrow\quad\operatorname{PP}=2 $$ **Final answer.** Perplexity = 2. **Question. Two language models score the same three-token test sequence. Model A gives next-token probabilities 0.5, 0.5, 0.5. Model B gives 0.5, 0.25, 0.25. Compute their perplexities and select the better intrinsic result.** #### Given Same sequence and N=3 for both models. A factors: 0.5,0.5,0.5. B factors: 0.5,0.25,0.25. #### To find Both sequence probabilities and perplexities. #### Formula and why it applies $$ P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3} $$ A fair comparison uses identical test tokens and scoring conventions. #### Solution ##### Step 1. Calculate A's probability $$ P_A=0.5\times0.5\times0.5=0.125=1/8 $$ ##### Step 2. Calculate A's perplexity $$ \operatorname{PP}_A=8^{1/3}=2 $$ ##### Step 3. Calculate B's probability $$ P_B=0.5\times0.25\times0.25=0.03125=1/32 $$ ##### Step 4. Calculate B's perplexity $$ \operatorname{PP}_B=32^{1/3}\approx3.174802 $$ ##### Step 5. Compare 2 is lower than 3.174802; A places more probability on the observed sequence. **Final answer.** Model A is better by perplexity on this test: 2 versus about 3.174802. **Question. The bigram training corpus is “This is a dog”, “This is a cat”, “I love my cat”, “This is my name”. Score “This is my cat”, including an end marker, and compute perplexity using five predicted events.** #### Given Each training sentence has one start and end marker. Test predictions: This, is, my, cat, . Thus N=5. #### To find Bigram sentence probability and perplexity. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5} $$ Start is given context. Four words plus the end marker produce five probability factors. #### Solution ##### Step 1. Count the required transitions | Transition | Pair count / history count | | --- | --- | | →This | 3/4 | | This→is | 3/3 | | is→my | 1/3 | | my→cat | 1/2 | | cat→ | 2/2 | ##### Step 2. Form the product $$ P=\frac34\times1\times\frac13\times\frac12\times1=\frac18 $$ ##### Step 3. Take the reciprocal $$ 1/P=8 $$ ##### Step 4. Take the fifth root $$ \operatorname{PP}=8^{1/5}\approx1.515717 $$ **Final answer.** P(W)=1/8 and perplexity≈1.515717 using five predicted events. **If the question explicitly states N = 6:** use that supplied convention and report PP = 81/6 ≈ 1.414214. The class slide counts both boundary markers in N. Write the chosen N beside the formula; do not silently switch between 5 and 6. Also, This occurs three times and is follows it three times, so P(is|This)=3/3, not 3/4. **Question. A uniform model assigns probability 1/10 to each actual next word in a five-token test. Find its perplexity.** #### Given Five scored tokens, each with assigned probability 0.1. N=5. #### To find The sentence probability and perplexity. #### Formula and why it applies $$ P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5} $$ Uniform equal probabilities make the effective number of choices especially clear. #### Solution ##### Step 1. Multiply the five factors $$ P(W)=0.1^5=0.00001=10^{-5} $$ ##### Step 2. Invert $$ 1/P(W)=10^5 $$ ##### Step 3. Take the fifth root $$ \operatorname{PP}=(10^5)^{1/5}=10 $$ **Final answer.** Perplexity = 10. **Yaad rakho:** Probability high → surprise low → perplexity low.