Natural Language Processing · TEE · papers

Q4a · Jaccard, cosine and Euclidean without preprocessing

2023-01-30 · re-exam · recovered AI paper

Previous examination · complete worked answers

Question. Explain Jaccard, cosine and Euclidean measures. Calculate them without preprocessing for D₁=“Data is the new oil of the digital economy” and D₂=“Data is a new oil”. [10 marks]

Answer

Given

Use exactly the given case and words; do not remove stopwords.

To find

Three measures in explicitly stated representations.

Formula: which one and why

J(A,B)=\frac{|A\cap B|}{|A\cup B|},\quad d_E(u,v)=\sqrt{\sum_i(u_i-v_i)^2}

Jaccard uses sets of distinct tokens. Cosine and Euclidean use raw-count BoW vectors in one shared order; do not secretly turn repeated the into a single count.

Step 1. Build the common vocabulary and counts

Document	Data	is	the	new	oil	of	digital	economy	a
D1	1	1	2	1	1	1	1	1	0
D2	1	1	0	1	1	0	0	0	1

Step 2. Jaccard sets

Intersection={Data,is,new,oil}, size4; union size9. J=4/9=0.444444; Jaccard distance=1−J=5/9.

Step 3. Cosine

Dot product=4. D1 squared length=1+1+4+1+1+1+1+1=11; D2 squared length=5. Cosine=4/√55=0.539359890.

Step 4. Euclidean

Coordinate differences: [0, 0, 2, 0, 0, 1, 1, 1, -1]. Sum of squares=8. Distance=√8=2.828427125. Smaller distance means greater proximity; this is a distance, not a probability.

Final answer

Jaccard=0.444444; cosine=0.539360; Euclidean distance=2.828427.