# Q2a · Six preprocessing operations

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q2a

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q2a/

Previous examination · complete worked answers

**Question. Explain lowercasing, tokenization, stopword removal, stemming, lemmatization and POS tagging with examples.** [6 marks]

### Answer

**Tokenization divides text into units called tokens for further processing.** A token may be a word, punctuation mark, character or subword, depending on the tokenizer. **Sentence segmentation** identifies sentence boundaries. Tokenization is not necessarily just splitting on spaces.

| Method | Example | Trade-off |
| --- | --- | --- |
| Word | “cats sleep” → [cats,sleep] | Easy to interpret; unfamiliar words can be unseen tokens |
| Character | cat → [c,a,t] | Handles spelling units; creates longer sequences |
| Subword | unhappiness → pieces such as [un,happi,ness] in an illustrative vocabulary | Balances vocabulary size and rare-word representation |
| Sentence | “Birds fly. Fish swim.” → two sentences | Abbreviations and punctuation complicate boundaries |

Actual subword splits depend on the learned vocabulary; the example is illustrative, not a universal tokenizer output. Word segmentation is especially important where spaces do not reliably separate words or in compound-rich languages.

**Question. Tokenize “Dr. Rao can't come today.” into words and punctuation under this rule: preserve Dr. and can't as whole tokens, and split only the final full stop.**

#### Given

The tokenization rule is explicitly supplied.

#### To find

The resulting token sequence.

#### Formula and why it applies

Apply the stated boundary rule; do not assume every period or apostrophe is a split point.

#### Step 1. Handle abbreviation

Dr. remains one token.

#### Step 2. Handle contraction

can't remains one token.

#### Step 3. Write sequence

[Dr., Rao, can't, come, today, .]

**Answer.** 6 tokens under the specified rule. Another tokenizer may legitimately produce a different sequence.

### Three difficult cases

**New York** may be two tokens but one named entity. **3.14** should not normally end a sentence at the decimal point. **don't** may remain whole or split into do + n't, depending on the tokenizer. Always state the convention when calculating counts.

Token = chosen unit. Space split ko universal rule mat samajh.

**Normalization reduces superficial variation so comparable forms can be processed consistently.** Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization.

**Stopwords** are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us.

**Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.**

#### Given

Operations and stopword set are fixed. Do not remove not.

#### To find

The final token list.

#### Formula and why it applies

Apply the requested operations in order, keeping a visible intermediate result after each transformation.

#### Step 1. Lowercase

the cats are not sleeping!!!

#### Step 2. Tokenize and remove punctuation

[the,cats,are,not,sleeping]

#### Step 3. Remove specified stopwords

Remove the and are → [cats,not,sleeping].

#### Step 4. Lemmatize

cats→cat; sleeping→sleep → [cat,not,sleep].

**Answer.** [cat,not,sleep]. The negative meaning is retained.

### Task decides preprocessing

For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them.

Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi.

**Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma.

| Criterion | Stemming | Lemmatization |
| --- | --- | --- |
| Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis |
| Output | May be nonword, e.g. studi | Dictionary form, e.g. study |
| Context/POS | Usually little or none | Often important |
| Irregular forms | May miss went→go | Can map went→go with verb analysis |
| Cost | Generally simpler | Usually more linguistic processing |

Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation.

### Three worked reductions

1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked.
2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation.
3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter.

**Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice.

Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan.

**Part-of-speech (POS) tagging** assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence.

| Tag | Role | Example in context |
| --- | --- | --- |
| NN / NNS | Singular / plural common noun | a book / two books |
| NNP | Proper noun | Mary, Mumbai |
| VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads |
| JJ | Adjective | the red book |
| RB | Adverb | runs quickly |
| DT | Determiner | the, a, this |
| IN | Preposition or subordinating conjunction | on the table |
| PRP | Personal pronoun | she, they |
| MD | Modal auxiliary | can, will, should |

**Open classes** readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. **Closed classes** contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes.

Input: Book that flight

→

Context: an imperative request

→

Output: Book/VB that/DT flight/NN

Compare “hand me that **book**”: book is a noun here. Similarly, “I **can** swim” has can/MD, while “the **can** is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is **POS ambiguity**.

| Sentence | Tagged interpretation | Reason |
| --- | --- | --- |
| They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. |
| The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. |
| She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. |
| Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. |

POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning.

**Yaad rakho:** Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai.