Natural Language Processing · TEE · papers
Q5a · Preprocess the NLP passage
2025-02-21 · re-exam · recovered AI paper
Previous examination · complete worked answers
Question. Perform the requested text preprocessing on: “Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through natural language. The ultimate objective of NLP is to read, decipher, understand, and make sense of human languages in a valuable way.” Remove the supplied stopwords {is,a,of,that,on,I,the,and,to,in}. [6 marks]
Answer
| Stage | Result |
|---|---|
| Sentence segmentation | Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through natural language. The ultimate objective of NLP is to read, decipher, understand, and make sense of human languages in a valuable way. |
| Word tokens (punctuation omitted) | Natural, Language, Processing, NLP, is, a, field, of, artificial, intelligence, that, focuses, on, the, interaction, between, computers, and, humans, through, natural, language, The, ultimate, objective, of, NLP, is, to, read, decipher, understand, and, make, sense, of, human, languages, in, a, valuable, way |
| Lowercase | natural, language, processing, nlp, is, a, field, of, artificial, intelligence, that, focuses, on, the, interaction, between, computers, and, humans, through, natural, language, the, ultimate, objective, of, nlp, is, to, read, decipher, understand, and, make, sense, of, human, languages, in, a, valuable, way |
| Declared stoplist | a, and, i, in, is, of, on, that, the, to |
| After exact stop removal | natural, language, processing, nlp, field, artificial, intelligence, focuses, interaction, between, computers, humans, through, natural, language, ultimate, objective, nlp, read, decipher, understand, make, sense, human, languages, valuable, way |
| Operation | Why it is useful |
|---|---|
| Sentence segmentation | Defines separate units for parsing and prevents counts from crossing sentence boundaries. |
| Lowercasing | Merges case variants when capitalization is not needed; retain original case for NER when useful. |
| Stop removal | Can remove frequent low-information words for a bag-of-words retrieval task; it is task-dependent and can damage negation/grammar. |
| Tokenization | Defines the units whose counts/features are computed; punctuation and abbreviation conventions must be consistent. |