Natural Language Processing · TEE · read

Generative versus discriminative models and Bayes' rule

POS tagging and its learning approaches

Unit 4, pages 3–46

Let x be the observed words and y be the desired tag sequence. A generative model describes their joint distribution P(x,y), often as P(y)P(x|y). A discriminative model learns P(y|x) directly or learns a decision boundary that separates possible labels. Both kinds can be trained with labeled data; “generative” does not mean “always unsupervised”.

LaTeX: P(y\mid x)=\frac{P(x\mid y)P(y)}{P(x)}

Question. A token has two possible tags, noun N and verb V. P(N)=0.6, P(V)=0.4, P(book|N)=0.2 and P(book|V)=0.5. Calculate P(N|book) and P(V|book).

Given

Two tags are exhaustive. Priors: 0.6,0.4. Word likelihoods: 0.2,0.5.

To find

Normalized posterior probabilities.

Formula and why it applies

LaTeX: P(t\mid w)=\frac{P(w\mid t)P(t)}{\sum_{u}P(w\mid u)P(u)}

Bayes' rule combines prior and likelihood. Normalization is needed to convert scores into posterior probabilities.

Solution

Step 1. Calculate the noun score

LaTeX: 0.2\times0.6=0.12

Step 2. Calculate the verb score

LaTeX: 0.5\times0.4=0.20

Step 3. Find the evidence

LaTeX: P(book)=0.12+0.20=0.32

Step 4. Normalize the noun score

LaTeX: P(N\mid book)=0.12/0.32=0.375

Step 5. Normalize the verb score

LaTeX: P(V\mid book)=0.20/0.32=0.625

Final answer. Verb is more probable in this simplified model: 0.625 versus 0.375.

Question. Two candidate tags have prior-likelihood products 0.06 and 0.14. Find their normalized probabilities.

Given

Unnormalized scores s₁=0.06 and s₂=0.14; these are the only candidates.

To find

The two posterior probabilities.

Formula and why it applies

LaTeX: P_i=s_i/(s_1+s_2)

A common denominator converts the candidate scores to a distribution summing to one.

Solution

Step 1. Add scores

LaTeX: 0.06+0.14=0.20

Step 2. Normalize first

LaTeX: 0.06/0.20=0.3

Step 3. Normalize second

LaTeX: 0.14/0.20=0.7

Final answer. Posterior probabilities are 0.3 and 0.7.

Yaad rakho: Generative: tag se word ka model. Discriminative: word se tag ka decision.