Natural Language Processing · TEE · formulas

Multinomial Naive Bayes: the exact solving method

WSD: Lesk & Naive Bayes

Formulas explained through full worked examples

Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison.

LaTeX: \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s).

For add-one multinomial smoothing:

LaTeX: P(w\mid s)=\frac{C(w,s)+1}{N_s+V}.

LaTeX: C(w,s) counts occurrences of word LaTeX: w in training contexts of sense LaTeX: s; LaTeX: N_s is the total number of word occurrences in those contexts; LaTeX: V is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors.

Always keep these counts separate

Class priors use number of examples. Word likelihoods in this multinomial model use number of word occurrences. The vocabulary counts unique words across all classes.

1. List the class examples and calculate priors.
2. Write one shared vocabulary and count its size.
3. Count total word occurrences in each class.
4. Count each query word in each class.
5. Apply the given smoothing rule.
6. Multiply each prior by all its query likelihoods.
7. Compare scores; divide each score by their sum only if a normalized posterior is requested.

LaTeX: P(s\mid C)=\frac{\operatorname{score}(s)}{\sum_{s'}\operatorname{score}(s')}.

Why smoothing is needed: an unseen clue gives LaTeX: 0/N_s=0; one zero makes the whole product zero. Add-one smoothing redistributes probability across every vocabulary word and prevents these zero factors. Use it only when requested, taught for the task, or state the assumption.

Different model warning: a Bernoulli Naive Bayes question counts presence/absence per document and may include absence factors. Do not use its denominator in a multinomial word-count question.

Pehle sentences se prior; phir words se likelihood; dono counts mix mat kar.