Natural Language Processing · M2 · formulas

Log probabilities: replace long products with sums

Smoothing & perplexity

Formulas explained through full worked examples

Probabilities are at most 1, so multiplying many of them produces very small numbers. Computers can round extremely small positive products to zero: numerical underflow. Log probabilities convert multiplication into addition and are more stable for long sequences. Natural logs use base e; log₂ uses base 2. Choose one base consistently.

\ln P(W)=\sum_{i=1}^{N}\ln p_iLaTeX: \ln P(W)=\sum_{i=1}^{N}\ln p_i
P(W)=e^{\ln P(W)},\qquad \operatorname{PP}(W)=\exp\left(-\frac1N\sum_i\ln p_i\right)LaTeX: P(W)=e^{\ln P(W)},\qquad \operatorname{PP}(W)=\exp\left(-\frac1N\sum_i\ln p_i\right)

The log of a probability below 1 is negative. A value closer to zero means a larger probability: ln(0.5) ≈ −0.693 is larger than ln(0.1) ≈ −2.303. A zero probability has log −∞; taking logs does not solve the unseen-event problem. Smoothing handles that separate problem.

Question. The next-word probabilities are 0.5, 0.25 and 0.5. Calculate the natural log probability, ordinary probability and perplexity.

Given

p₁=0.5; p₂=0.25; p₃=0.5; N=3.

To find

ln P(W), P(W), and PP.

Formula and why it applies

\ln P=\sum_i\ln p_i,\quad P=e^{\ln P},\quad\operatorname{PP}=e^{-\ln P/N}LaTeX: \ln P=\sum_i\ln p_i,\quad P=e^{\ln P},\quad\operatorname{PP}=e^{-\ln P/N}

Logarithms preserve products as sums and the exponent converts the result back.

Solution

Step 1. Compute individual logs
\ln0.5\approx-0.693147,\quad\ln0.25\approx-1.386294LaTeX: \ln0.5\approx-0.693147,\quad\ln0.25\approx-1.386294
Step 2. Add all three terms
\ln P=-0.693147-1.386294-0.693147\approx-2.772589LaTeX: \ln P=-0.693147-1.386294-0.693147\approx-2.772589
Step 3. Recover the probability
P=e^{-2.772589}\approx0.0625LaTeX: P=e^{-2.772589}\approx0.0625
Step 4. Find average negative log probability
-\ln P/3\approx2.772589/3=0.924196LaTeX: -\ln P/3\approx2.772589/3=0.924196
Step 5. Exponentiate
\operatorname{PP}=e^{0.924196}\approx2.519842LaTeX: \operatorname{PP}=e^{0.924196}\approx2.519842
Final answer. ln P≈−2.772589, P=0.0625, PP≈2.519842.

Question. A model has a total natural log probability −12 over six scored tokens. Calculate its perplexity.

Given

ln P(W)=−12; N=6.

To find

PP without forming the tiny ordinary probability.

Formula and why it applies

\operatorname{PP}=\exp(-\ln P(W)/N)LaTeX: \operatorname{PP}=\exp(-\ln P(W)/N)

The logarithmic formula directly uses the supplied log score.

Solution

Step 1. Negate the log score
-\ln P=12LaTeX: -\ln P=12
Step 2. Divide by token count
12/6=2LaTeX: 12/6=2
Step 3. Apply exponential
\operatorname{PP}=e^2\approx7.389056LaTeX: \operatorname{PP}=e^2\approx7.389056
Final answer. Perplexity≈7.389056.

Question. A language model's average negative log₂ probability is 3 bits per token. Find its perplexity.

Given

Cross-entropy H=3 bits/token. Log base is 2.

To find

PP.

Formula and why it applies

\operatorname{PP}=2^HLaTeX: \operatorname{PP}=2^H

The inverse operation of log₂ is raising 2 to that power; use e only for natural logs.

Solution

Step 1. Identify the log base

Bits mean base 2.

Step 2. Raise the base to H
\operatorname{PP}=2^3=8LaTeX: \operatorname{PP}=2^3=8
Final answer. Perplexity = 8.

Yaad rakho: ln use kiya toh e se wapas; log₂ use kiya toh 2 se wapas.