Natural Language Processing · TEE · papers
Q2a · Six preprocessing operations
2024-12-03 · final · recovered AI paper
Previous examination · complete worked answers
Answer
Tokenization divides text into units called tokens for further processing. A token may be a word, punctuation mark, character or subword, depending on the tokenizer. Sentence segmentation identifies sentence boundaries. Tokenization is not necessarily just splitting on spaces.
| Method | Example | Trade-off |
|---|---|---|
| Word | “cats sleep” → [cats,sleep] | Easy to interpret; unfamiliar words can be unseen tokens |
| Character | cat → [c,a,t] | Handles spelling units; creates longer sequences |
| Subword | unhappiness → pieces such as [un,happi,ness] in an illustrative vocabulary | Balances vocabulary size and rare-word representation |
| Sentence | “Birds fly. Fish swim.” → two sentences | Abbreviations and punctuation complicate boundaries |
Actual subword splits depend on the learned vocabulary; the example is illustrative, not a universal tokenizer output. Word segmentation is especially important where spaces do not reliably separate words or in compound-rich languages.
Question. Tokenize “Dr. Rao can't come today.” into words and punctuation under this rule: preserve Dr. and can't as whole tokens, and split only the final full stop.
Given
The tokenization rule is explicitly supplied.
To find
The resulting token sequence.
Formula and why it applies
Apply the stated boundary rule; do not assume every period or apostrophe is a split point.
Step 1. Handle abbreviation
Dr. remains one token.
Step 2. Handle contraction
can't remains one token.
Step 3. Write sequence
[Dr., Rao, can't, come, today, .]
Answer. 6 tokens under the specified rule. Another tokenizer may legitimately produce a different sequence.
Three difficult cases
New York may be two tokens but one named entity. 3.14 should not normally end a sentence at the decimal point. don't may remain whole or split into do + n't, depending on the tokenizer. Always state the convention when calculating counts.
Token = chosen unit. Space split ko universal rule mat samajh.
Normalization reduces superficial variation so comparable forms can be processed consistently. Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization.
Stopwords are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us.
Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.
Given
Operations and stopword set are fixed. Do not remove not.
To find
The final token list.
Formula and why it applies
Apply the requested operations in order, keeping a visible intermediate result after each transformation.
Step 1. Lowercase
the cats are not sleeping!!!
Step 2. Tokenize and remove punctuation
[the,cats,are,not,sleeping]
Step 3. Remove specified stopwords
Remove the and are → [cats,not,sleeping].
Step 4. Lemmatize
cats→cat; sleeping→sleep → [cat,not,sleep].
Answer. [cat,not,sleep]. The negative meaning is retained.
Task decides preprocessing
For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them.
Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi.
Stemming applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. Lemmatization uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma.
| Criterion | Stemming | Lemmatization |
|---|---|---|
| Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis |
| Output | May be nonword, e.g. studi | Dictionary form, e.g. study |
| Context/POS | Usually little or none | Often important |
| Irregular forms | May miss went→go | Can map went→go with verb analysis |
| Cost | Generally simpler | Usually more linguistic processing |
Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation.
Three worked reductions
- studies, studying: an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked.
- better: adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation.
- saw: as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter.
Over-stemming merges forms that should remain distinct; under-stemming fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice.
Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan.
Part-of-speech (POS) tagging assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence.
| Tag | Role | Example in context |
|---|---|---|
| NN / NNS | Singular / plural common noun | a book / two books |
| NNP | Proper noun | Mary, Mumbai |
| VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads |
| JJ | Adjective | the red book |
| RB | Adverb | runs quickly |
| DT | Determiner | the, a, this |
| IN | Preposition or subordinating conjunction | on the table |
| PRP | Personal pronoun | she, they |
| MD | Modal auxiliary | can, will, should |
Open classes readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. Closed classes contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes.
Compare “hand me that book”: book is a noun here. Similarly, “I can swim” has can/MD, while “the can is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is POS ambiguity.
| Sentence | Tagged interpretation | Reason |
|---|---|---|
| They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. |
| The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. |
| She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. |
| Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. |
POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning.
Yaad rakho: Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai.