# Q2b · Text vectorization alternatives

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q2b

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q2b/

Previous examination · complete worked answers

**Question. Define word embeddings and explain one-hot, Bag of Words, bag of n-grams and TF–IDF with advantages and disadvantages.** [10 marks]

### Answer

**A sparse representation** has many zero entries and often one coordinate per vocabulary item. BoW and TF–IDF are common sparse document representations. **A dense embedding** has comparatively few learned real-valued coordinates; one word's properties are spread across many coordinates.

| Representation | What coordinates mean | Strength | Limitation |
| --- | --- | --- | --- |
| One-hot | One identity position per vocabulary word | Unambiguous ID | Different words are orthogonal; no learned similarity |
| Count-based context vector | Context-word co-occurrence counts | Transparent corpus evidence | Large, sparse; frequent words may dominate |
| TF–IDF document vector | Weighted vocabulary terms | Useful retrieval weighting | Word order and contextual meaning are limited |
| Dense word embedding | Learned latent features | Compact semantic/syntactic relationships | Coordinates are not generally human-labelled features |

**Distributional** describes learning from patterns of linguistic context. **Distributed** describes representing an item across multiple coordinates rather than a single active identity bit. These are related but different ideas. A one-hot ID alone does not learn distributional similarity.

### Model design choices

Choose contexts (document, sentence, fixed window), weighting (raw counts, normalization, TF–IDF-like weighting or association scores), dimensionality reduction or learned embeddings, and a similarity measure. Cosine compares direction and is less affected by length than a raw dot product; Euclidean distance also reflects magnitude.

Sparse = bahut zero. Dense = chhoti learned number-list. Similar context se useful similarity seekhte hain.

| Method | Construction and example | Strength | Limitation |
| --- | --- | --- | --- |
| One-hot | One1in vocabulary coordinate: cat=(1,0,0) | Simple unique identity | Sparse; all distinct words equally distant |
| BoW | Counts per vocabulary coordinate | Simple interpretable frequency | Loses order and context |
| Bag of n-grams | Counts contiguous word groups, e.g. not good | Retains local order/phrases | Vocabulary grows; sparse unseen phrases |
| TF–IDF | Term frequency weighted by inverse document frequency | Downweights ubiquitous terms for retrieval | No contextual meaning; depends on corpus statistics |

Dense learned embeddings differ from these sparse count encodings: similarity may reflect learned context. Explain the representation before computing any metric.