# Q3b · Stemming and lemmatization

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q3b

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q3b/

Previous examination · complete worked answers

**Question. Discuss the need for stemming, list different types of stemming, and differentiate stemming from lemmatization.** [10 marks]

### Answer

Stemming merges inflected/derived surface variants for matching and reduces vocabulary size. Types include rule/affix-removal stemmers (Porter, Snowball, Lancaster), dictionary/lookup stemmers, statistical stemmers and hybrid methods. More aggressive reduction increases both merging and overstemming risk.

**Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma.

| Criterion | Stemming | Lemmatization |
| --- | --- | --- |
| Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis |
| Output | May be nonword, e.g. studi | Dictionary form, e.g. study |
| Context/POS | Usually little or none | Often important |
| Irregular forms | May miss went→go | Can map went→go with verb analysis |
| Cost | Generally simpler | Usually more linguistic processing |

Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation.

### Three worked reductions

1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked.
2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation.
3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter.

**Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice.

Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan.