Natural Language Processing · M2 · mocks

Q3 (3 marks)

M2 Mock C · mixed numerical practice

M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice

Q3

For trigram prediction, C(i like)=5, C(i like tea)=3 and C(i like coffee)=2. Find the MLE probability of each continuation. Then find add-one P(tea|i like) using a continuation vocabulary of size 4. (3 marks)

Given

Two-word history i like occurs 5 times; continuations tea=3, coffee=2; |V|=4.

To find

Two MLE values and the smoothed tea probability.

Formula and why it applies

P(w_i\mid w_{i-2},w_{i-1})=\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})}LaTeX: P(w_i\mid w_{i-2},w_{i-1})=\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})}
P_{+1}(w\mid h)=\frac{C(hw)+1}{C(h)+|V|}LaTeX: P_{+1}(w\mid h)=\frac{C(hw)+1}{C(h)+|V|}

A trigram conditions on two preceding words. The same add-one count rule applies to that longer history.

Solution

Step 1. Tea MLE

P(\text{tea}\mid i,\text{like})=\frac35=0.6LaTeX: P(\text{tea}\mid i,\text{like})=\frac35=0.6

Step 2. Coffee MLE

P(\text{coffee}\mid i,\text{like})=\frac25=0.4LaTeX: P(\text{coffee}\mid i,\text{like})=\frac25=0.4

Step 3. Smooth tea

P_{+1}(\text{tea}\mid i,\text{like})=\frac{3+1}{5+4}=\frac49\approx0.444444LaTeX: P_{+1}(\text{tea}\mid i,\text{like})=\frac{3+1}{5+4}=\frac49\approx0.444444
MLE tea=0.6; coffee=0.4; smoothed tea=4/9≈0.444444.