Natural Language Processing · TEE · papers

Q1a · Normalization changes vocabulary

2023-01-30 · re-exam · recovered AI paper

Previous examination · complete worked answers

Question. “The vocabulary of a corpus remains the same before and after normalization.” State true or false and justify with an example. [5 marks]

Answer

False in general. Vocabulary is the set of distinct tokens under a chosen tokenizer. In “Cats cats running”, raw case-sensitive vocabulary={Cats,cats,running}, size3. Lowercase merges Cats/cats: {cats,running}, size2. Lemmatization may yield {cat,run}; stopword removal can remove entries. For an already normalized corpus the set may happen to remain unchanged, so the correct claim is that normalization can change it, not that it always must.