Natural Language Processing · TEE · read-chapter

Text preprocessing, normalization and lexical diversity

TEE · Unit 2 PDF pages 1–50

  1. Tokenization: sentence, word, character and subword
  2. Cleaning, case folding, punctuation and stopwords
  3. Stemming versus lemmatization
  4. Type–token ratio: three fully worked sums