# Q6b · VSM, TF and IDF in retrieval

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q6b

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q6b/

Previous examination · complete worked answers

**Question. Define Vector Space Model, Term Frequency and Inverse Document Frequency with reference to information retrieval.** [10 marks]

### Answer

**A vocabulary** is the ordered collection of distinct terms allowed by a representation. Its order defines what every coordinate means. A **vector** is simply an ordered number-list. A **document–term matrix** puts one document in each row and one vocabulary term in each column.

With vocabulary [cat,dog,fish], the count vector [2,0,1] means cat appears twice, dog zero times and fish once. With [dog,cat,fish], the same text becomes [0,2,1]. A vector without its vocabulary order is incomplete.

| Document | cat | dog | fish |
| --- | --- | --- | --- |
| cat cat fish | 2 | 0 | 1 |
| dog fish | 0 | 1 | 1 |

**Nominal categories** have no inherent order, such as cities. **Ordinal categories** do have an order, such as low < medium < high. Assigning arbitrary word IDs does not create meaningful numerical distances between words.

Vector ka har box ek fixed word ka hai. Pehle column order fix, phir numbers.

**TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus.** The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency:



$$
\operatorname{TF}(t,d)=\frac{C(t,d)}{N_d}
$$





$$
\operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)}
$$





$$
\operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t).
$$





$$
N_d
$$

 is the document's token count after the specified preprocessing; 

$$
D
$$

 is number of documents; 

$$
\operatorname{df}(t)
$$

 is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency.

**Why the log?** It compresses the ratio's scale. If a term occurs in all documents, 

$$
D/df=1
$$

, so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default.

**Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.**

#### Given

D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10.

#### To find

The TF–IDF weight.

#### Formula and why it applies



$$
TF=C/N_d
$$

, 

$$
IDF=\log_{10}(D/df)
$$

, then multiply.

#### Step 1. TF



$$
3/12=0.25
$$

.

#### Step 2. Document ratio



$$
10/2=5
$$

.

#### Step 3. IDF



$$
\log_{10}5\approx0.698970
$$

.

#### Step 4. Weight



$$
0.25\times0.698970\approx0.174743
$$

.

**Answer.** TF=0.25, IDF≈0.698970, TF–IDF≈0.174743.

**Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.**

#### Given

C=4, Nd=20, D=8, df=8.

#### To find

Its weight under unsmoothed base-10 IDF.

#### Formula and why it applies



$$
TF=C/N_d
$$

, 

$$
IDF=\log_{10}(D/df)
$$

.

#### Step 1. TF



$$
4/20=0.2
$$

.

#### Step 2. IDF



$$
\log_{10}(8/8)=\log_{10}1=0
$$

.

#### Step 3. Multiply



$$
0.2\times0=0
$$

.

**Answer.** Weight 0: this term does not distinguish documents under this convention.

**Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.**

#### Given

No further preprocessing. Each term occurs in exactly two of the three documents.

#### To find

All three vectors.

#### Formula and why it applies

First count df by documents, then compute each document's TF, then multiply by its term's IDF.

#### Step 1. Common IDF



$$
D=3
$$

, 

$$
df(cat)=df(dog)=df(fish)=2
$$

. Each IDF 

$$
=\log_{10}(3/2)\approx0.176091
$$

.

#### Step 2. D1 term frequencies

Length 3 → 

$$
[2/3,1/3,0]
$$

.

#### Step 3. D1 weights



$$
[(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0]
$$

.

#### Step 4. D2 weights

Length 2; TF 

$$
[0,1/2,1/2]
$$

. Vector 

$$
\approx[0,0.088046,0.088046]
$$

.

#### Step 5. D3 weights

Length 3; TF 

$$
[1/3,0,2/3]
$$

. Vector 

$$
\approx[0.058697,0,0.117394]
$$

.

**Answer.** D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394].

TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply.

**Information retrieval (IR)** locates and ranks items relevant to an information need, usually expressed as a query. The collection may contain documents, product descriptions or scientific papers. Retrieval estimates relevance; it does not itself establish that every retrieved claim is true.

Documents → cleaning

→

Build index

→

Query → same representation

→

Match & rank

→

Return relevant results

### The inverted index

An inverted index maps a term to the documents containing it, often with positions or counts. For D1 “cats drink milk” and D2 “dogs drink water”, the posting list for drink is {D1,D2}, while milk maps only to D1. This avoids rescanning every document for every query.

| Term | Posting list |
| --- | --- |
| cats | D1 |
| drink | D1, D2 |
| milk | D1 |
| dogs | D2 |
| water | D2 |

### Matching and ranking

Boolean retrieval combines term membership using AND, OR and NOT. Vector-space retrieval represents documents and queries as weighted vectors, for example with TF-IDF, and compares them with cosine similarity. Learned retrieval can use dense embeddings for meaning-based matching. The representation used at query time must be compatible with the document index.

### Search engines and recommendation

A search engine receives an explicit query and ranks matching indexed pages. A recommender may derive an implicit query from a user’s liked items or profile and retrieve similar candidates. Both require candidate retrieval and ranking, but the source of the information need differs.

### Evaluate the results

For retrieval, TP is the number of retrieved relevant documents, FP retrieved irrelevant documents, and FN relevant documents not retrieved. Precision and recall use the same formulas as classification. Ranking measures also consider the positions of relevant items; a relevant result at rank 1 is more useful than one far below.

### Retrieval evaluation

A search system retrieves 8 documents; 6 are relevant. There are 10 relevant documents in the collection. Find precision, recall and F1.

**Given**

Retrieved=8; relevant retrieved=6; all relevant=10.

**To find**

P, R and F1.

### Formula and why it applies



$$
P=\frac{\text{relevant retrieved}}{\text{retrieved}},\quad R=\frac{\text{relevant retrieved}}{\text{all relevant}}
$$





$$
F_1=\frac{2PR}{P+R}
$$



The denominator is 8 for precision and 10 for recall because they ask different questions.

### Solution

#### Step 1. Identify the confusion counts

TP=6; FP=8−6=2; FN=10−6=4.

#### Step 2. Precision



$$
P=\frac68=0.75
$$



#### Step 3. Recall



$$
R=\frac6{10}=0.6
$$



#### Step 4. F1



$$
F_1=\frac{2(0.75)(0.6)}{0.75+0.6}=\frac{0.9}{1.35}=\frac23\approx0.666667
$$



Precision=0.75, recall=0.60, F1≈0.666667.

**Yaad rakho:** Precision: jo laaye, unmein kitne sahi? Recall: jo sahi the, unmein kitne laaye?