Natural Language Processing · TEE · papers
Q3b · Preprocessing pipeline
2024-02-12 · re-exam · recovered AI paper
Previous examination · complete worked answers
Answer
An NLP pipeline is an organized sequence of operations that transforms raw language into a task output and checks whether that output is useful. Not every task needs every cleaning operation.
- Data acquisition: collect relevant text and, when needed, labels.
- Cleaning and preprocessing: handle markup, tokenize, normalize and decide which information to retain.
- Feature engineering: represent text numerically using BoW, TF–IDF or embeddings.
- Modelling: apply rules or train a statistical/neural model.
- Evaluation: test on held-out data using task-appropriate measures.
- Deployment: make the system available, log suitable feedback and monitor errors or data changes.
Concrete spam-filter example
Collect labelled emails → separate train/test data → tokenize text while preserving meaningful terms → fit vocabulary and IDF on training data → train classifier → evaluate spam detection on the held-out set → deploy an email service filter → monitor legitimate emails wrongly blocked.
Intrinsic evaluation measures model/task quality, such as precision, recall or tagging accuracy. Extrinsic evaluation measures effect in the wider application, such as whether users find answers faster. A higher internal metric does not automatically guarantee a better user outcome.
Data leakage: learning vocabulary statistics, labels or preprocessing decisions from the test set can make reported performance too optimistic. Fit learned transformations on training data, then apply them to validation/test data.
Collect → clean → numbers → model → test → use. Test data ko training ki information mat banao.
Normalization reduces superficial variation so comparable forms can be processed consistently. Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization.
Stopwords are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us.
Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.
Given
Operations and stopword set are fixed. Do not remove not.
To find
The final token list.
Formula and why it applies
Apply the requested operations in order, keeping a visible intermediate result after each transformation.
Step 1. Lowercase
the cats are not sleeping!!!
Step 2. Tokenize and remove punctuation
[the,cats,are,not,sleeping]
Step 3. Remove specified stopwords
Remove the and are → [cats,not,sleeping].
Step 4. Lemmatize
cats→cat; sleeping→sleep → [cat,not,sleep].
Answer. [cat,not,sleep]. The negative meaning is retained.
Task decides preprocessing
For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them.
Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi.
Stemming applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. Lemmatization uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma.
| Criterion | Stemming | Lemmatization |
|---|---|---|
| Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis |
| Output | May be nonword, e.g. studi | Dictionary form, e.g. study |
| Context/POS | Usually little or none | Often important |
| Irregular forms | May miss went→go | Can map went→go with verb analysis |
| Cost | Generally simpler | Usually more linguistic processing |
Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation.
Three worked reductions
- studies, studying: an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked.
- better: adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation.
- saw: as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter.
Over-stemming merges forms that should remain distinct; under-stemming fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice.
Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan.