Natural Language Processing · TEE · papers
Q4b · Three-document TF–IDF
2023-12-01 · final · recovered AI paper
Previous examination · complete worked answers
Question. Calculate TF–IDF for: (1) Inflation increased unemployment (2) Company increased sales (3) Fear increased pulse. [10 marks]
Answer
Given
| Document | Original text |
|---|---|
| 1 | Inflation increased unemployment |
| 2 | Company increased sales |
| 3 | Fear increased pulse |
Stopwords: none supplied
To find
BoW, IDF and TF–IDF
Formula: which one and why
Use lowercase alphabetic tokens and exactly the stated stoplist. If no stoplist is supplied, retain function words. Adopt raw-count TF and natural-log IDF; the paper does not prescribe another TF/log convention.
LaTeX: \mathrm{TFIDF}(w,d)=c(w,d)\ln\frac{N}{\mathrm{df}(w)}A term shared by all N documents has IDF ln(1)=0. Counts and TF–IDF weights are different representations.
Step 1. Tokenize each document
| Document | Tokens |
|---|---|
| 1 | inflation, increased, unemployment |
| 2 | company, increased, sales |
| 3 | fear, increased, pulse |
Step 2. Fix the vocabulary order
company, fear, increased, inflation, pulse, sales, unemployment
Step 3. Count occurrences (Bag of Words)
| Document | company | fear | increased | inflation | pulse | sales | unemployment |
|---|---|---|---|---|---|---|---|
| 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 |
| 2 | 1 | 0 | 1 | 0 | 0 | 1 | 0 |
| 3 | 0 | 1 | 1 | 0 | 1 | 0 | 0 |
Step 4. Calculate document frequency and IDF
| Term | df | N / df | ln(N/df) |
|---|---|---|---|
| company | 1 | 3 | 1.098612289 |
| fear | 1 | 3 | 1.098612289 |
| increased | 3 | 1 | 0.000000000 |
| inflation | 1 | 3 | 1.098612289 |
| pulse | 1 | 3 | 1.098612289 |
| sales | 1 | 3 | 1.098612289 |
| unemployment | 1 | 3 | 1.098612289 |
Step 5. Multiply each TF by its IDF
| Document | company | fear | increased | inflation | pulse | sales | unemployment |
|---|---|---|---|---|---|---|---|
| 1 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | 1.098612 |
| 2 | 1.098612 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 0.000000 |
| 3 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | 1.098612 | 0.000000 | 0.000000 |
Final answer
The final table is the TF–IDF matrix in the declared vocabulary order. A different explicitly stated log base rescales all IDFs; length-normalized TF changes each document scale.