# Q2a · TF–IDF cosine after a fixed stoplist

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q2a

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q2a/

Previous examination · complete worked answers

**Question. For “The cat sat on the mat.”, “The dog barked at the cat.” and “I love my dog and my cat.” remove stopwords {I,the,on,my,at,and}; construct TF–IDF and calculate cosine similarity between the first two documents.** [10 marks]

### Answer

### Given

| Document | Original text |
| --- | --- |
| 1 | The cat sat on the mat. |
| 2 | The dog barked at the cat. |
| 3 | I love my dog and my cat. |

Stopwords: at, my, i, on, the, and

### To find

BoW, IDF and TF–IDF plus requested cosine

### Formula: which one and why

Use lowercase alphabetic tokens and exactly the stated stoplist. If no stoplist is supplied, retain function words. Adopt raw-count TF and natural-log IDF; the paper does not prescribe another TF/log convention.



$$
\mathrm{TFIDF}(w,d)=c(w,d)\ln\frac{N}{\mathrm{df}(w)}
$$



A term shared by all N documents has IDF ln(1)=0. Counts and TF–IDF weights are different representations.

### Step 1. Tokenize each document

| Document | Tokens |
| --- | --- |
| 1 | cat, sat, mat |
| 2 | dog, barked, cat |
| 3 | love, dog, cat |

### Step 2. Fix the vocabulary order

barked, cat, dog, love, mat, sat

### Step 3. Count occurrences (Bag of Words)

| Document | barked | cat | dog | love | mat | sat |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 0 | 1 | 0 | 0 | 1 | 1 |
| 2 | 1 | 1 | 1 | 0 | 0 | 0 |
| 3 | 0 | 1 | 1 | 1 | 0 | 0 |

### Step 4. Calculate document frequency and IDF

| Term | df | N / df | ln(N/df) |
| --- | --- | --- | --- |
| barked | 1 | 3 | 1.098612289 |
| cat | 3 | 1 | 0.000000000 |
| dog | 2 | 3/2 | 0.405465108 |
| love | 1 | 3 | 1.098612289 |
| mat | 1 | 3 | 1.098612289 |
| sat | 1 | 3 | 1.098612289 |

### Step 5. Multiply each TF by its IDF

| Document | barked | cat | dog | love | mat | sat |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 1.098612 |
| 2 | 1.098612 | 0.000000 | 0.405465 | 0.000000 | 0.000000 | 0.000000 |
| 3 | 0.000000 | 0.000000 | 0.405465 | 1.098612 | 0.000000 | 0.000000 |

### Step 6. Calculate cosine from these same TF–IDF rows



$$
\cos(u,v)=\frac{\sum_i u_i v_i}{\sqrt{\sum_i u_i^2}\sqrt{\sum_i v_i^2}}
$$



Coordinate products: 0.000000000, 0.000000000, 0.000000000, 0.000000000, 0.000000000, 0.000000000

Dot product=0.000000000; squared norms=2.413897922, 1.371350915. Cosine=0.000000000

### Final answer

The final table is the TF–IDF matrix in the declared vocabulary order. A different explicitly stated log base rescales all IDFs; length-normalized TF changes each document scale.

Cosine is 0.0. Shared words whose IDF is zero contribute nothing to the TF–IDF dot product.

### Which terms are prominent?

D1: sat and mat have IDF ln3; D2: barked has ln3 and dog has ln(3/2); D3: love has ln3 and dog has ln(3/2). Cat occurs in all three documents and has zero unsmoothed IDF. Thus shared cat does not make D1/D2 TF–IDF cosine positive.