# Log probabilities: replace long products with sums

Subject: Natural Language Processing

Exam: M2

Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-smoothing/log-probabilities

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-smoothing--log-probabilities/

Formulas explained through full worked examples

Probabilities are at most 1, so multiplying many of them produces very small numbers. Computers can round extremely small positive products to zero: **numerical underflow**. Log probabilities convert multiplication into addition and are more stable for long sequences. Natural logs use base e; log₂ uses base 2. Choose one base consistently.



$$
\ln P(W)=\sum_{i=1}^{N}\ln p_i
$$





$$
P(W)=e^{\ln P(W)},\qquad \operatorname{PP}(W)=\exp\left(-\frac1N\sum_i\ln p_i\right)
$$



The log of a probability below 1 is negative. A value closer to zero means a larger probability: ln(0.5) ≈ −0.693 is larger than ln(0.1) ≈ −2.303. A zero probability has log −∞; taking logs does not solve the unseen-event problem. Smoothing handles that separate problem.

**Question. The next-word probabilities are 0.5, 0.25 and 0.5. Calculate the natural log probability, ordinary probability and perplexity.**

#### Given

p₁=0.5; p₂=0.25; p₃=0.5; N=3.

#### To find

ln P(W), P(W), and PP.

#### Formula and why it applies



$$
\ln P=\sum_i\ln p_i,\quad P=e^{\ln P},\quad\operatorname{PP}=e^{-\ln P/N}
$$



Logarithms preserve products as sums and the exponent converts the result back.

#### Solution

##### Step 1. Compute individual logs



$$
\ln0.5\approx-0.693147,\quad\ln0.25\approx-1.386294
$$



##### Step 2. Add all three terms



$$
\ln P=-0.693147-1.386294-0.693147\approx-2.772589
$$



##### Step 3. Recover the probability



$$
P=e^{-2.772589}\approx0.0625
$$



##### Step 4. Find average negative log probability



$$
-\ln P/3\approx2.772589/3=0.924196
$$



##### Step 5. Exponentiate



$$
\operatorname{PP}=e^{0.924196}\approx2.519842
$$



**Final answer.** ln P≈−2.772589, P=0.0625, PP≈2.519842.

**Question. A model has a total natural log probability −12 over six scored tokens. Calculate its perplexity.**

#### Given

ln P(W)=−12; N=6.

#### To find

PP without forming the tiny ordinary probability.

#### Formula and why it applies



$$
\operatorname{PP}=\exp(-\ln P(W)/N)
$$



The logarithmic formula directly uses the supplied log score.

#### Solution

##### Step 1. Negate the log score



$$
-\ln P=12
$$



##### Step 2. Divide by token count



$$
12/6=2
$$



##### Step 3. Apply exponential



$$
\operatorname{PP}=e^2\approx7.389056
$$



**Final answer.** Perplexity≈7.389056.

**Question. A language model's average negative log₂ probability is 3 bits per token. Find its perplexity.**

#### Given

Cross-entropy H=3 bits/token. Log base is 2.

#### To find

PP.

#### Formula and why it applies



$$
\operatorname{PP}=2^H
$$



The inverse operation of log₂ is raising 2 to that power; use e only for natural logs.

#### Solution

##### Step 1. Identify the log base

Bits mean base 2.

##### Step 2. Raise the base to H



$$
\operatorname{PP}=2^3=8
$$



**Final answer.** Perplexity = 8.

**Yaad rakho:** ln use kiya toh e se wapas; log₂ use kiya toh 2 se wapas.