Natural Language Processing · TEE · formulas

TF–IDF from the exact slide convention

Representation & similarity

Formulas explained through full worked examples

TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus. The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency:

\operatorname{TF}(t,d)=\frac{C(t,d)}{N_d}

\operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)}

\operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t).

$N_d$ is the document's token count after the specified preprocessing; $D$ is number of documents; $\operatorname{df}(t)$ is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency.

Why the log? It compresses the ratio's scale. If a term occurs in all documents, $D/df=1$ , so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default.

Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.

Given

D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10.

To find

The TF–IDF weight.

Formula and why it applies

$TF=C/N_d$ , $IDF=\log_{10}(D/df)$ , then multiply.

Step 1. TF

$3/12=0.25$ .

Step 2. Document ratio

$10/2=5$ .

Step 3. IDF

$\log_{10}5\approx0.698970$ .

Step 4. Weight

$0.25\times0.698970\approx0.174743$ .

Answer. TF=0.25, IDF≈0.698970, TF–IDF≈0.174743.

Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.

Given

C=4, Nd=20, D=8, df=8.

To find

Its weight under unsmoothed base-10 IDF.

Formula and why it applies

$TF=C/N_d$ , $IDF=\log_{10}(D/df)$ .

Step 1. TF

$4/20=0.2$ .

Step 2. IDF

$\log_{10}(8/8)=\log_{10}1=0$ .

Step 3. Multiply

$0.2\times0=0$ .

Answer. Weight 0: this term does not distinguish documents under this convention.

Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.

Given

No further preprocessing. Each term occurs in exactly two of the three documents.

To find

All three vectors.

Formula and why it applies

First count df by documents, then compute each document's TF, then multiply by its term's IDF.

Step 1. Common IDF

$D=3$ , $df(cat)=df(dog)=df(fish)=2$ . Each IDF $=\log_{10}(3/2)\approx0.176091$ .

Step 2. D1 term frequencies

Length 3 → $[2/3,1/3,0]$ .

Step 3. D1 weights

$[(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0]$ .

Step 4. D2 weights

Length 2; TF $[0,1/2,1/2]$ . Vector $\approx[0,0.088046,0.088046]$ .

Step 5. D3 weights

Length 3; TF $[1/3,0,2/3]$ . Vector $\approx[0.058697,0,0.117394]$ .

Answer. D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394].

TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply.