Natural Language Processing · M2 · mocks

Q3 (3 marks)

M2 Mock B · unseen pairs & decoding

M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice

Q3

For a fixed vocabulary of five possible continuations, C(very)=7 and C(very bright)=0. Calculate the unsmoothed and Laplace-smoothed probabilities of bright after very. (3 marks)

Given

History count=7; pair count=0; |V|=5.

To find

MLE and add-one probabilities.

Formula and why it applies

P_{MLE}(w\mid h)=\frac{C(hw)}{C(h)}LaTeX: P_{MLE}(w\mid h)=\frac{C(hw)}{C(h)}
P_{+1}(w\mid h)=\frac{C(hw)+1}{C(h)+|V|}LaTeX: P_{+1}(w\mid h)=\frac{C(hw)+1}{C(h)+|V|}

MLE uses only observations. Laplace reserves probability for all known vocabulary outcomes, including unseen pairs.

Solution

Step 1. Unsmoothed estimate

P_{MLE}(\text{bright}\mid\text{very})=\frac07=0LaTeX: P_{MLE}(\text{bright}\mid\text{very})=\frac07=0

Step 2. Add one to the pair count

0+1=1LaTeX: 0+1=1

Step 3. Add vocabulary size to the history count

7+5=12LaTeX: 7+5=12

Step 4. Divide

P_{+1}(\text{bright}\mid\text{very})=\frac1{12}\approx0.083333LaTeX: P_{+1}(\text{bright}\mid\text{very})=\frac1{12}\approx0.083333
MLE=0; Laplace=1/12≈0.083333. The zero count means unseen in this corpus, not impossible language.