# TF–IDF from the exact slide convention

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u2-tfidf

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u2-tfidf/

Formulas explained through full worked examples

**TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus.** The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency:



$$
\operatorname{TF}(t,d)=\frac{C(t,d)}{N_d}
$$





$$
\operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)}
$$





$$
\operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t).
$$





$$
N_d
$$

 is the document's token count after the specified preprocessing; 

$$
D
$$

 is number of documents; 

$$
\operatorname{df}(t)
$$

 is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency.

**Why the log?** It compresses the ratio's scale. If a term occurs in all documents, 

$$
D/df=1
$$

, so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default.

**Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.**

#### Given

D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10.

#### To find

The TF–IDF weight.

#### Formula and why it applies



$$
TF=C/N_d
$$

, 

$$
IDF=\log_{10}(D/df)
$$

, then multiply.

#### Step 1. TF



$$
3/12=0.25
$$

.

#### Step 2. Document ratio



$$
10/2=5
$$

.

#### Step 3. IDF



$$
\log_{10}5\approx0.698970
$$

.

#### Step 4. Weight



$$
0.25\times0.698970\approx0.174743
$$

.

**Answer.** TF=0.25, IDF≈0.698970, TF–IDF≈0.174743.

**Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.**

#### Given

C=4, Nd=20, D=8, df=8.

#### To find

Its weight under unsmoothed base-10 IDF.

#### Formula and why it applies



$$
TF=C/N_d
$$

, 

$$
IDF=\log_{10}(D/df)
$$

.

#### Step 1. TF



$$
4/20=0.2
$$

.

#### Step 2. IDF



$$
\log_{10}(8/8)=\log_{10}1=0
$$

.

#### Step 3. Multiply



$$
0.2\times0=0
$$

.

**Answer.** Weight 0: this term does not distinguish documents under this convention.

**Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.**

#### Given

No further preprocessing. Each term occurs in exactly two of the three documents.

#### To find

All three vectors.

#### Formula and why it applies

First count df by documents, then compute each document's TF, then multiply by its term's IDF.

#### Step 1. Common IDF



$$
D=3
$$

, 

$$
df(cat)=df(dog)=df(fish)=2
$$

. Each IDF 

$$
=\log_{10}(3/2)\approx0.176091
$$

.

#### Step 2. D1 term frequencies

Length 3 → 

$$
[2/3,1/3,0]
$$

.

#### Step 3. D1 weights



$$
[(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0]
$$

.

#### Step 4. D2 weights

Length 2; TF 

$$
[0,1/2,1/2]
$$

. Vector 

$$
\approx[0,0.088046,0.088046]
$$

.

#### Step 5. D3 weights

Length 3; TF 

$$
[1/3,0,2/3]
$$

. Vector 

$$
\approx[0.058697,0,0.117394]
$$

.

**Answer.** D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394].

TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply.