Natural Language Processing · M2 · read

Collocations and the purpose of a language model

N-grams and language models

Unit 3, pages 62–108; essential probability revision

A collocation is a conventional word combination whose words occur together more strongly than ordinary chance or independent frequencies would suggest. Examples include “strong tea”, “heavy rain” and “red wine”. A bigram is merely two consecutive items; it need not be a collocation. “of the” can be frequent because both words are frequent, while a collocation expresses a particularly strong association.

Definition. A statistical language model assigns a probability to a sequence of words, or equivalently predicts the next word from preceding context. It is useful for autocomplete, speech recognition, machine translation, spelling correction, assistive communication and ranking candidate text. High probability reflects patterns learned from its training distribution; it is not a guarantee of truth or grammatical correctness.

Noisy-channel connection. Suppose x is the observed noisy text or speech and y is a candidate intended message. The noisy-channel approach chooses the y that maximizes P(x | y)P(y). The channel model P(x | y) asks how y could produce x; the language model P(y) asks whether y is a plausible message. P(x) is the same for all candidate y and therefore can be omitted when ranking.

LaTeX: \hat y=\arg\max_y P(y\mid x)=\arg\max_y P(x\mid y)P(y)

Question. A spelling-correction system considers “flowers” and “flours”. For the observed input, the channel likelihoods are 0.2 and 0.4, and the respective language-model probabilities in context are 0.08 and 0.01. Which candidate does the noisy-channel rule choose?

Given

Candidate flowers: P(x|flowers)=0.2, P(flowers)=0.08. Candidate flours: P(x|flours)=0.4, P(flours)=0.01.

To find

The larger unnormalized candidate score.

Formula and why it applies

LaTeX: \operatorname{score}(y)=P(x\mid y)P(y)

Both the error mechanism and the language context matter. The same denominator would normalize both candidates.

Solution

Step 1. Score flowers

LaTeX: 0.2\times0.08=0.016

Step 2. Score flours

LaTeX: 0.4\times0.01=0.004

Step 3. Compare

LaTeX: 0.016>0.004

Final answer. Choose flowers. The scores are not yet posterior probabilities.

Yaad rakho: Har word-pair bigram hai; har frequent pair meaningful collocation nahi.