Natural Language Processing · TEE · papers
Q3a · Text preprocessing
2022-11-22 · final · recovered AI paper
Previous examination · complete worked answers
Answer
Tokenization divides text into units called tokens for further processing. A token may be a word, punctuation mark, character or subword, depending on the tokenizer. Sentence segmentation identifies sentence boundaries. Tokenization is not necessarily just splitting on spaces.
| Method | Example | Trade-off |
|---|---|---|
| Word | “cats sleep” → [cats,sleep] | Easy to interpret; unfamiliar words can be unseen tokens |
| Character | cat → [c,a,t] | Handles spelling units; creates longer sequences |
| Subword | unhappiness → pieces such as [un,happi,ness] in an illustrative vocabulary | Balances vocabulary size and rare-word representation |
| Sentence | “Birds fly. Fish swim.” → two sentences | Abbreviations and punctuation complicate boundaries |
Actual subword splits depend on the learned vocabulary; the example is illustrative, not a universal tokenizer output. Word segmentation is especially important where spaces do not reliably separate words or in compound-rich languages.
Question. Tokenize “Dr. Rao can't come today.” into words and punctuation under this rule: preserve Dr. and can't as whole tokens, and split only the final full stop.
Given
The tokenization rule is explicitly supplied.
To find
The resulting token sequence.
Formula and why it applies
Apply the stated boundary rule; do not assume every period or apostrophe is a split point.
Step 1. Handle abbreviation
Dr. remains one token.
Step 2. Handle contraction
can't remains one token.
Step 3. Write sequence
[Dr., Rao, can't, come, today, .]
Answer. 6 tokens under the specified rule. Another tokenizer may legitimately produce a different sequence.
Three difficult cases
New York may be two tokens but one named entity. 3.14 should not normally end a sentence at the decimal point. don't may remain whole or split into do + n't, depending on the tokenizer. Always state the convention when calculating counts.
Token = chosen unit. Space split ko universal rule mat samajh.
Normalization reduces superficial variation so comparable forms can be processed consistently. Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization.
Stopwords are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us.
Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.
Given
Operations and stopword set are fixed. Do not remove not.
To find
The final token list.
Formula and why it applies
Apply the requested operations in order, keeping a visible intermediate result after each transformation.
Step 1. Lowercase
the cats are not sleeping!!!
Step 2. Tokenize and remove punctuation
[the,cats,are,not,sleeping]
Step 3. Remove specified stopwords
Remove the and are → [cats,not,sleeping].
Step 4. Lemmatize
cats→cat; sleeping→sleep → [cat,not,sleep].
Answer. [cat,not,sleep]. The negative meaning is retained.
Task decides preprocessing
For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them.
Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi.
Stemming applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. Lemmatization uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma.
| Criterion | Stemming | Lemmatization |
|---|---|---|
| Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis |
| Output | May be nonword, e.g. studi | Dictionary form, e.g. study |
| Context/POS | Usually little or none | Often important |
| Irregular forms | May miss went→go | Can map went→go with verb analysis |
| Cost | Generally simpler | Usually more linguistic processing |
Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation.
Three worked reductions
- studies, studying: an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked.
- better: adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation.
- saw: as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter.
Over-stemming merges forms that should remain distinct; under-stemming fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice.
Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan.