Natural Language Processing · TEE · papers
Q7B · Gutenberg corpus and normalization
Solved re-exam · February 2026
Actual re-examination · 9 February 2026 · 100 marks · 3 hours
ii. Explain the importance of text normalization in Natural Language Processing (NLP). Illustrate your answer with a suitable example of an application where text normalization plays a crucial role. [10 marks]
Answer
i. Using the Gutenberg corpus
The Gutenberg corpus provides literary texts, commonly used through tools such as NLTK’s Gutenberg collection. It supplies naturally occurring sentences and longer documents for exploring language patterns. The available subset and licensing should be checked before reuse in a new distribution.
| Task | Use of the corpus |
|---|---|
| Frequency and vocabulary study | Count tokens and word types, inspect common words and compare lexical diversity across texts |
| Concordances and collocations | View a word in context and discover words that occur near each other |
| N-gram language modeling | Estimate unigram or bigram counts and compare sequence probabilities |
| Stylistic or authorship analysis | Compare sentence lengths, vocabulary and function-word patterns across authors |
| Text preprocessing experiments | Study the effect of tokenization, case normalization and lemmatization on counts |
A sound workflow selects files, inspects encoding and boundaries, tokenizes consistently, computes the relevant statistics and interprets them against corpus composition. Literary text is not automatically representative of chat, medical records or present-day search queries; domain mismatch matters.
ii. Text normalization
Normalization converts selected surface variants into a consistent form so that equivalent expressions do not unnecessarily produce different features. Possible steps include case folding, whitespace cleanup, Unicode normalization, contraction expansion and lemmatization. Choose these according to the task.
Example: product search
- Raw query: “ RUNNING shoes ”. Product description: “Comfortable run shoes”.
- Trim extra whitespace and apply consistent case: “running shoes”.
- If the application chooses lemmatization, running → run. The query and the indexed description now share the intended run and shoes features.
- Apply the same normalization to stored documents and future queries, otherwise matching remains inconsistent.
Normalization reduces avoidable vocabulary variation and can improve recall. It must preserve distinctions important to the task: lowercasing US to us may lose meaning, and removing “not” can reverse sentiment. Aggressive stemming or synonym replacement can create incorrect matches.
Yaad rakho: Same meaning ke surface variants ko consistent banao; useful meaning delete mat karo.