Natural Language Processing · TEE · papers

Q4a · NLP pipeline

2025-02-21 · re-exam · recovered AI paper

Previous examination · complete worked answers

Answer

An NLP pipeline is an organized sequence of operations that transforms raw language into a task output and checks whether that output is useful. Not every task needs every cleaning operation.

1. Data acquisition: collect relevant text and, when needed, labels.
2. Cleaning and preprocessing: handle markup, tokenize, normalize and decide which information to retain.
3. Feature engineering: represent text numerically using BoW, TF–IDF or embeddings.
4. Modelling: apply rules or train a statistical/neural model.
5. Evaluation: test on held-out data using task-appropriate measures.
6. Deployment: make the system available, log suitable feedback and monitor errors or data changes.

Concrete spam-filter example

Collect labelled emails → separate train/test data → tokenize text while preserving meaningful terms → fit vocabulary and IDF on training data → train classifier → evaluate spam detection on the held-out set → deploy an email service filter → monitor legitimate emails wrongly blocked.

Intrinsic evaluation measures model/task quality, such as precision, recall or tagging accuracy. Extrinsic evaluation measures effect in the wider application, such as whether users find answers faster. A higher internal metric does not automatically guarantee a better user outcome.

Data leakage: learning vocabulary statistics, labels or preprocessing decisions from the test set can make reported performance too optimistic. Fit learned transformations on training data, then apply them to validation/test data.

Collect → clean → numbers → model → test → use. Test data ko training ki information mat banao.

Definition. NLP is the study of computational methods for analyzing, understanding and generating human language. The pipeline diagram above turns raw language into a task-specific prediction or generated response.