Natural Language Processing · TEE · papers
Q3b · Stemming and lemmatization
2023-12-01 · final · recovered AI paper
Previous examination · complete worked answers
Answer
Stemming merges inflected/derived surface variants for matching and reduces vocabulary size. Types include rule/affix-removal stemmers (Porter, Snowball, Lancaster), dictionary/lookup stemmers, statistical stemmers and hybrid methods. More aggressive reduction increases both merging and overstemming risk.
Stemming applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. Lemmatization uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma.
| Criterion | Stemming | Lemmatization |
|---|---|---|
| Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis |
| Output | May be nonword, e.g. studi | Dictionary form, e.g. study |
| Context/POS | Usually little or none | Often important |
| Irregular forms | May miss went→go | Can map went→go with verb analysis |
| Cost | Generally simpler | Usually more linguistic processing |
Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation.
Three worked reductions
- studies, studying: an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked.
- better: adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation.
- saw: as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter.
Over-stemming merges forms that should remain distinct; under-stemming fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice.
Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan.