Natural Language Processing · M2 · formulas

Laplace smoothing: why both numerator and denominator change

Smoothing & perplexity

Formulas explained through full worked examples

Smoothing reallocates some probability mass to events that received zero or unreliable estimates from finite training data. Laplace, or add-one, smoothing adds one pseudo-count to every possible next outcome. If there are V possible outcomes, the total added count is V, so adding 1 only to the numerator would produce probabilities that do not sum to 1.

LaTeX: P_{\mathrm{Laplace}}(w)=\frac{C(w)+1}{N+V}

LaTeX: P_{\mathrm{Laplace}}(w\mid h)=\frac{C(h,w)+1}{C(h)+V}

For unigram smoothing, N is the total training token count. For bigram smoothing, C(h) is the count of the particular history. In both cases V is the number of outcomes being predicted, not the number of training sentences and not the number of distinct bigrams. When end is a possible prediction, include it in V; the start marker is usually context only.

Question. A fixed unigram vocabulary is {a,b,c}. The training counts are 3,1,0. Compute unsmoothed and Laplace-smoothed probabilities.

Given

C(a)=3, C(b)=1, C(c)=0; N=4; V=3.

To find

The complete distributions.

Formula and why it applies

LaTeX: P(w)=C(w)/N,\qquad P_L(w)=(C(w)+1)/(N+V)

The zero-count c is still in the fixed vocabulary, so it must receive one pseudo-count and contribute to V.

Solution

Step 1. Compute unsmoothed values

Word	Unsmoothed
a	3/4 = 0.75
b	1/4 = 0.25
c	0/4 = 0

Step 2. Calculate the new denominator

LaTeX: N+V=4+3=7

Step 3. Add one to each numerator

LaTeX: C(a)+1=4,\quad C(b)+1=2,\quad C(c)+1=1

Step 4. Divide

Word	Smoothed
a	4/7 ≈ 0.571429
b	2/7 ≈ 0.285714
c	1/7 ≈ 0.142857

Step 5. Confirm the total

LaTeX: 4/7+2/7+1/7=1

Final answer. Smoothed probabilities: a=4/7, b=2/7, c=1/7. Common a loses mass; unseen c gains mass.

Question. After history I, a corpus records like three times, study once and sleep zero times. The allowed next vocabulary is {like,study,sleep}. Compute all Laplace-smoothed bigram probabilities.

Given

C(I,like)=3; C(I,study)=1; C(I,sleep)=0. History count C(I)=4. V=3.

To find

P_L(like|I), P_L(study|I), P_L(sleep|I).

Formula and why it applies

LaTeX: P_L(w\mid I)=\frac{C(I,w)+1}{C(I)+V}

This is smoothing within the I row; use its total 4, not the global corpus token count.

Solution

Step 1. Form the common denominator

LaTeX: C(I)+V=4+3=7

Step 2. Calculate like

LaTeX: P_L(like\mid I)=(3+1)/7=4/7

Step 3. Calculate study

LaTeX: P_L(study\mid I)=(1+1)/7=2/7

Step 4. Calculate sleep

LaTeX: P_L(sleep\mid I)=(0+1)/7=1/7

Final answer. The row is (4/7, 2/7, 1/7). An unseen bigram is no longer forced to zero.

Question. A fixed vocabulary has five possible next tokens. A history h is unseen in training. What is its add-one probability for each next token?

Given

C(h)=0; C(h,w)=0 for each outcome; V=5.

To find

P_L(w|h) for each token.

Formula and why it applies

LaTeX: P_L(w\mid h)=\frac{0+1}{0+V}

All outcomes receive equal pseudo-counts because the training data gives no preference for this history.

Solution

Step 1. Add the pseudo-count

LaTeX: 0+1=1

Step 2. Form the denominator

LaTeX: 0+5=5

Step 3. Divide and sum

LaTeX: P_L(w\mid h)=1/5=0.2,\qquad5(0.2)=1

Final answer. Every outcome gets 0.2 under this fixed-vocabulary model.

Vocabulary warning. Add-one does not automatically define probabilities for every word in the world. Unknown words require a declared vocabulary/UNK convention. If a question supplies V, use that stated V rather than inventing additional tokens.

Yaad rakho: Add-one: upar +1, neeche +V; bigram mein neeche history ka count.