Natural Language Processing · TEE · about

Add-k: preserve a normalized vocabulary row

Optional extensions — after the TEE questions

Additional explanation; no corresponding question in the two supplied TEE papers. No exam marks are implied.

Optional extension. Add-k is a supporting generalization of Laplace smoothing. The supplied TEE papers ask add-one bigram sentence probabilities, not an add-k calculation. Complete the actual solved papers first.

Add-k smoothing generalizes add-one smoothing. Add the same positive pseudo-count k to each allowed outcome. There are V outcomes, so the total pseudo-count is kV. Setting k=1 recovers Laplace smoothing. A smaller k usually changes the original counts less; it is not automatically the best choice for every dataset.

P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV},\qquad k>0.LaTeX: P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV},\qquad k>0.

Use one fixed vocabulary, including an explicit unknown-word policy if needed. Choose k from training/validation evidence, not by fitting the final test results. Unigram smoothing uses N, the total token count, in place of C(h).

Question. After history h, counts for the complete next-token vocabulary [a,b,c] are [3,1,0]. Calculate add-k probabilities for k=0.5 and check the row sum. [Authored worked example]

Answer

Given

C(h)=4; V=3; k=1/2; the zero-count word c is an allowed outcome.

To find

All three probabilities and their interpretation.

Formula and why it applies

P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV}LaTeX: P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV}

Smoothing must adjust the denominator for every pseudo-count added across the row.

Solution

Step 1. Form the new total
4+(0.5)(3)=5.5.LaTeX: 4+(0.5)(3)=5.5.
Step 2. Calculate a
P(a\mid h)=\frac{3+0.5}{5.5}=\frac7{11}\approx0.636364.LaTeX: P(a\mid h)=\frac{3+0.5}{5.5}=\frac7{11}\approx0.636364.
Step 3. Calculate b and c separately
P(b\mid h)=\frac{1+0.5}{5.5}=\frac3{11}\approx0.272727.LaTeX: P(b\mid h)=\frac{1+0.5}{5.5}=\frac3{11}\approx0.272727.
P(c\mid h)=\frac{0+0.5}{5.5}=\frac1{11}\approx0.090909.LaTeX: P(c\mid h)=\frac{0+0.5}{5.5}=\frac1{11}\approx0.090909.
Step 4. Check normalization
\frac7{11}+\frac3{11}+\frac1{11}=1.LaTeX: \frac7{11}+\frac3{11}+\frac1{11}=1.
Final answer. The row is [7/11,3/11,1/11]. The unseen allowed outcome c receives positive probability, and the row remains a probability distribution.

This worked lesson was added after the video plan. No matching video has been generated or reviewed.