Natural Language Processing · TEE · papers

Q2b · Text vectorization alternatives

2023-01-30 · re-exam · recovered AI paper

Previous examination · complete worked answers

Question. Define word embeddings and explain one-hot, Bag of Words, bag of n-grams and TF–IDF with advantages and disadvantages. [10 marks]

Answer

A sparse representation has many zero entries and often one coordinate per vocabulary item. BoW and TF–IDF are common sparse document representations. A dense embedding has comparatively few learned real-valued coordinates; one word's properties are spread across many coordinates.

Representation	What coordinates mean	Strength	Limitation
One-hot	One identity position per vocabulary word	Unambiguous ID	Different words are orthogonal; no learned similarity
Count-based context vector	Context-word co-occurrence counts	Transparent corpus evidence	Large, sparse; frequent words may dominate
TF–IDF document vector	Weighted vocabulary terms	Useful retrieval weighting	Word order and contextual meaning are limited
Dense word embedding	Learned latent features	Compact semantic/syntactic relationships	Coordinates are not generally human-labelled features

Distributional describes learning from patterns of linguistic context. Distributed describes representing an item across multiple coordinates rather than a single active identity bit. These are related but different ideas. A one-hot ID alone does not learn distributional similarity.

Model design choices

Choose contexts (document, sentence, fixed window), weighting (raw counts, normalization, TF–IDF-like weighting or association scores), dimensionality reduction or learned embeddings, and a similarity measure. Cosine compares direction and is less affected by length than a raw dot product; Euclidean distance also reflects magnitude.

Sparse = bahut zero. Dense = chhoti learned number-list. Similar context se useful similarity seekhte hain.

Method	Construction and example	Strength	Limitation
One-hot	One1in vocabulary coordinate: cat=(1,0,0)	Simple unique identity	Sparse; all distinct words equally distant
BoW	Counts per vocabulary coordinate	Simple interpretable frequency	Loses order and context
Bag of n-grams	Counts contiguous word groups, e.g. not good	Retains local order/phrases	Vocabulary grows; sparse unseen phrases
TF–IDF	Term frequency weighted by inverse document frequency	Downweights ubiquitous terms for retrieval	No contextual meaning; depends on corpus statistics

Dense learned embeddings differ from these sparse count encodings: similarity may reflect learned context. Explain the representation before computing any metric.