Natural Language Processing · TEE · read

Small probabilities, logs and calculator use

Start here · language & probability

Shared language and probability foundations · TEE

Repeated multiplication of probabilities produces small numbers. A calculator may show 0.0000125 as 1.25 × 10⁻⁵; they are the same value. A logarithm turns a product into a sum. Log probabilities are often negative because probabilities between 0 and 1 have negative logs.

LaTeX: \log(ab)=\log a+\log b

LaTeX: \log_b x=y\quad\Longleftrightarrow\quad b^y=x

Use the log base specified in the question. Natural log is ln (base e); ordinary calculator log is usually base 10. Perplexity is simplest to evaluate using the direct power formula. Do not mix bases in the same calculation.

Calculator example · roots in perplexity

A model assigns probability 1/16 to a test sequence with 4 predicted tokens. Calculate perplexity.

Given

P = 1/16; N = 4 predicted tokens.

To find

Perplexity PP.

Formula and why it applies

LaTeX: PP=P^{-1/N}=\left(\frac1P\right)^{1/N}

Perplexity takes the reciprocal probability and then its Nth root. N must match the number of predicted events in the product.

Solution

Step 1. Take the reciprocal

LaTeX: \frac{1}{1/16}=16

Step 2. Take the fourth root

LaTeX: PP=16^{1/4}=2

Step 3. Interpret

A perplexity of 2 means an effective average uncertainty comparable to two equally likely alternatives. It is not 2% error.

PP = 2.

Calculator entry order

Enter the full product with parentheses. For the Nth root enter (1 ÷ P)^(1 ÷ N). Keep at least 6 significant digits while working; round the final answer. If your calculator offers Ans, reuse the stored value instead of typing a heavily rounded result.

Yaad rakho: Probability ke factors multiply; log mein add; perplexity mein ulta karke root.