# Text preprocessing, normalization and lexical diversity

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-preprocess

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read-chapter/u2-preprocess/

TEE · Unit 2 PDF pages 1–50

1. [Tokenization: sentence, word, character and subword](https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-tokenize/)
2. [Cleaning, case folding, punctuation and stopwords](https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-normalization/)
3. [Stemming versus lemmatization](https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-stem-lemma/)
4. [Type–token ratio: three fully worked sums](https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-ttr/)