# Add-k: preserve a normalized vocabulary row

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/about/optional-extensions/add-k

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/about/optional-extensions--add-k/

Additional explanation; no corresponding question in the two supplied TEE papers. No exam marks are implied.

**Optional extension.** Add-k is a supporting generalization of Laplace smoothing. The supplied TEE papers ask add-one bigram sentence probabilities, not an add-k calculation. Complete the actual solved papers first.

**Add-k smoothing generalizes add-one smoothing.** Add the same positive pseudo-count k to each allowed outcome. There are V outcomes, so the total pseudo-count is kV. Setting k=1 recovers Laplace smoothing. A smaller k usually changes the original counts less; it is not automatically the best choice for every dataset.



$$
P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV},\qquad k>0.
$$



Use one fixed vocabulary, including an explicit unknown-word policy if needed. Choose k from training/validation evidence, not by fitting the final test results. Unigram smoothing uses N, the total token count, in place of C(h).

**Question. After history h, counts for the complete next-token vocabulary [a,b,c] are [3,1,0]. Calculate add-k probabilities for k=0.5 and check the row sum. [Authored worked example]**

### Answer

#### Given

C(h)=4; V=3; k=1/2; the zero-count word c is an allowed outcome.

#### To find

All three probabilities and their interpretation.

#### Formula and why it applies



$$
P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV}
$$



Smoothing must adjust the denominator for every pseudo-count added across the row.

#### Solution

##### Step 1. Form the new total



$$
4+(0.5)(3)=5.5.
$$



##### Step 2. Calculate a



$$
P(a\mid h)=\frac{3+0.5}{5.5}=\frac7{11}\approx0.636364.
$$



##### Step 3. Calculate b and c separately



$$
P(b\mid h)=\frac{1+0.5}{5.5}=\frac3{11}\approx0.272727.
$$





$$
P(c\mid h)=\frac{0+0.5}{5.5}=\frac1{11}\approx0.090909.
$$



##### Step 4. Check normalization



$$
\frac7{11}+\frac3{11}+\frac1{11}=1.
$$



**Final answer.** The row is [7/11,3/11,1/11]. The unseen allowed outcome c receives positive probability, and the row remains a probability distribution.

This worked lesson was added after the video plan. No matching video has been generated or reviewed.