Natural Language Processing · TEE · papers

Q4A · Three preprocessing techniques

Solved final paper · December 2025

Actual final examination · 2 December 2025 · 100 marks · 3 hours

Answer

Text preprocessing transforms raw text into a representation suitable for a chosen NLP task. The best steps depend on the task; removing information indiscriminately can reduce performance.

1. Tokenization

Tokenization divides text into units called tokens, such as words, subwords or sentences. These units are required for counting words and building many model inputs. “The cats are sleeping.” can become [The, cats, are, sleeping, .] under a punctuation-preserving word tokenizer. A different tokenizer may split contractions or punctuation differently, so the convention must be consistent during training and prediction.

2. Lowercasing

Lowercasing maps uppercase letters to lowercase so that variants such as “Book”, “BOOK” and “book” can share one feature. This reduces vocabulary duplication in a case-insensitive search system. However, “US” and “us” are different, and capitals help NER recognise names. Therefore lowercasing should be chosen according to the application.

3. Lemmatization

Lemmatization maps an inflected word to its dictionary base form, often using its POS and context. Examples: cats → cat; running/VBG → run; better/JJR → good. It allows a search for “run” to match relevant inflections. A stemmer instead uses simpler truncation rules and can produce a non-word stem; lemmatization is linguistically informed and can need more processing.

Combined example

These steps do not automatically remove stop words or punctuation. Those are separate choices. For sentiment analysis, preserve negation such as “not”; for syntax parsing, function words and word order are often essential.

Yaad rakho: Preprocess task dekh ke karo: har word ko blindly delete mat karo.