# Laplace smoothing: why both numerator and denominator change

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-smoothing/laplace-basics

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-smoothing--laplace-basics/

Formulas explained through full worked examples

**Smoothing** reallocates some probability mass to events that received zero or unreliable estimates from finite training data. **Laplace, or add-one, smoothing** adds one pseudo-count to every possible next outcome. If there are V possible outcomes, the total added count is V, so adding 1 only to the numerator would produce probabilities that do not sum to 1.



$$
P_{\mathrm{Laplace}}(w)=\frac{C(w)+1}{N+V}
$$





$$
P_{\mathrm{Laplace}}(w\mid h)=\frac{C(h,w)+1}{C(h)+V}
$$



For unigram smoothing, N is the total training token count. For bigram smoothing, C(h) is the count of the particular history. In both cases V is the number of outcomes being predicted, not the number of training sentences and not the number of distinct bigrams. When end is a possible prediction, include it in V; the start marker is usually context only.

Observed counts: 3, 1, 0

→

Add one to every outcome: 4, 2, 1

→

New total: 7

→

Smoothed probabilities: 4/7, 2/7, 1/7

**Question. A fixed unigram vocabulary is {a,b,c}. The training counts are 3,1,0. Compute unsmoothed and Laplace-smoothed probabilities.**

#### Given

C(a)=3, C(b)=1, C(c)=0; N=4; V=3.

#### To find

The complete distributions.

#### Formula and why it applies



$$
P(w)=C(w)/N,\qquad P_L(w)=(C(w)+1)/(N+V)
$$



The zero-count c is still in the fixed vocabulary, so it must receive one pseudo-count and contribute to V.

#### Solution

##### Step 1. Compute unsmoothed values

| Word | Unsmoothed |
| --- | --- |
| a | 3/4 = 0.75 |
| b | 1/4 = 0.25 |
| c | 0/4 = 0 |

##### Step 2. Calculate the new denominator



$$
N+V=4+3=7
$$



##### Step 3. Add one to each numerator



$$
C(a)+1=4,\quad C(b)+1=2,\quad C(c)+1=1
$$



##### Step 4. Divide

| Word | Smoothed |
| --- | --- |
| a | 4/7 ≈ 0.571429 |
| b | 2/7 ≈ 0.285714 |
| c | 1/7 ≈ 0.142857 |

##### Step 5. Confirm the total



$$
4/7+2/7+1/7=1
$$



**Final answer.** Smoothed probabilities: a=4/7, b=2/7, c=1/7. Common a loses mass; unseen c gains mass.

**Question. After history I, a corpus records like three times, study once and sleep zero times. The allowed next vocabulary is {like,study,sleep}. Compute all Laplace-smoothed bigram probabilities.**

#### Given

C(I,like)=3; C(I,study)=1; C(I,sleep)=0. History count C(I)=4. V=3.

#### To find

P\_L(like|I), P\_L(study|I), P\_L(sleep|I).

#### Formula and why it applies



$$
P_L(w\mid I)=\frac{C(I,w)+1}{C(I)+V}
$$



This is smoothing within the I row; use its total 4, not the global corpus token count.

#### Solution

##### Step 1. Form the common denominator



$$
C(I)+V=4+3=7
$$



##### Step 2. Calculate like



$$
P_L(like\mid I)=(3+1)/7=4/7
$$



##### Step 3. Calculate study



$$
P_L(study\mid I)=(1+1)/7=2/7
$$



##### Step 4. Calculate sleep



$$
P_L(sleep\mid I)=(0+1)/7=1/7
$$



**Final answer.** The row is (4/7, 2/7, 1/7). An unseen bigram is no longer forced to zero.

**Question. A fixed vocabulary has five possible next tokens. A history h is unseen in training. What is its add-one probability for each next token?**

#### Given

C(h)=0; C(h,w)=0 for each outcome; V=5.

#### To find

P\_L(w|h) for each token.

#### Formula and why it applies



$$
P_L(w\mid h)=\frac{0+1}{0+V}
$$



All outcomes receive equal pseudo-counts because the training data gives no preference for this history.

#### Solution

##### Step 1. Add the pseudo-count



$$
0+1=1
$$



##### Step 2. Form the denominator



$$
0+5=5
$$



##### Step 3. Divide and sum



$$
P_L(w\mid h)=1/5=0.2,\qquad5(0.2)=1
$$



**Final answer.** Every outcome gets 0.2 under this fixed-vocabulary model.

**Vocabulary warning.** Add-one does not automatically define probabilities for every word in the world. Unknown words require a declared vocabulary/UNK convention. If a question supplies V, use that stated V rather than inventing additional tokens.

**Yaad rakho:** Add-one: upar +1, neeche +V; bigram mein neeche history ka count.