Natural Language Processing · TEE · papers

Q2a · TF–IDF cosine after a fixed stoplist

2025-02-21 · re-exam · recovered AI paper

Previous examination · complete worked answers

Question. For “The cat sat on the mat.”, “The dog barked at the cat.” and “I love my dog and my cat.” remove stopwords {I,the,on,my,at,and}; construct TF–IDF and calculate cosine similarity between the first two documents. [10 marks]

Answer

Given

Document	Original text
1	The cat sat on the mat.
2	The dog barked at the cat.
3	I love my dog and my cat.

Stopwords: at, my, i, on, the, and

To find

BoW, IDF and TF–IDF plus requested cosine

Formula: which one and why

Use lowercase alphabetic tokens and exactly the stated stoplist. If no stoplist is supplied, retain function words. Adopt raw-count TF and natural-log IDF; the paper does not prescribe another TF/log convention.

\mathrm{TFIDF}(w,d)=c(w,d)\ln\frac{N}{\mathrm{df}(w)}

A term shared by all N documents has IDF ln(1)=0. Counts and TF–IDF weights are different representations.

Step 1. Tokenize each document

Document	Tokens
1	cat, sat, mat
2	dog, barked, cat
3	love, dog, cat

Step 2. Fix the vocabulary order

barked, cat, dog, love, mat, sat

Step 3. Count occurrences (Bag of Words)

Document	barked	cat	dog	love	mat	sat
1	0	1	0	0	1	1
2	1	1	1	0	0	0
3	0	1	1	1	0	0

Step 4. Calculate document frequency and IDF

Term	df	N / df	ln(N/df)
barked	1	3	1.098612289
cat	3	1	0.000000000
dog	2	3/2	0.405465108
love	1	3	1.098612289
mat	1	3	1.098612289
sat	1	3	1.098612289

Step 5. Multiply each TF by its IDF

Document	barked	dog	love	mat	sat
1	0.000000	0.000000	0.000000	1.098612	1.098612
2	1.098612	0.405465	0.000000	0.000000	0.000000
3	0.000000	0.405465	1.098612	0.000000	0.000000

Step 6. Calculate cosine from these same TF–IDF rows

\cos(u,v)=\frac{\sum_i u_i v_i}{\sqrt{\sum_i u_i^2}\sqrt{\sum_i v_i^2}}

Coordinate products: 0.000000000, 0.000000000, 0.000000000, 0.000000000, 0.000000000, 0.000000000

Dot product=0.000000000; squared norms=2.413897922, 1.371350915. Cosine=0.000000000

Final answer

The final table is the TF–IDF matrix in the declared vocabulary order. A different explicitly stated log base rescales all IDFs; length-normalized TF changes each document scale.

Cosine is 0.0. Shared words whose IDF is zero contribute nothing to the TF–IDF dot product.

Which terms are prominent?

D1: sat and mat have IDF ln3; D2: barked has ln3 and dog has ln(3/2); D3: love has ln3 and dog has ln(3/2). Cat occurs in all three documents and has zero unsmoothed IDF. Thus shared cat does not make D1/D2 TF–IDF cosine positive.