Natural Language Processing · TEE · papers
Q5a · Feature engineering, BoW and Euclidean
2024-12-03 · final · recovered AI paper
Previous examination · complete worked answers
Question. Explain text feature engineering and list vectorization approaches. For the three sentences below remove stopwords {is,that,are,to,and}, create vocabulary and BoW, then compute Euclidean distance between sentences1and2: “Natural Language Processing is fascinating”; “I love Natural Language processing and like to study Natural Language”; “I like things that are fascinating”. [10 marks]
Answer
Feature engineering converts text into informative model inputs. Common approaches: one-hot, BoW, bag of n-grams, TF–IDF and dense embeddings. This question specifically requests count-based BoW, so do not use TF–IDF in its distance.
Given
Use lowercase lexical tokens and the exact printed stoplist; I remains because it is not a stopword here.
To find
Vocabulary, count matrix and distance D1–D2.
Formula: which one and why
LaTeX: d_E(u,v)=\sqrt{\sum_i(u_i-v_i)^2}Step 1. Tokenize and remove the exact stoplist
| Document | Retained tokens |
|---|---|
| 1 | natural, language, processing, fascinating |
| 2 | i, love, natural, language, processing, like, study, natural, language |
| 3 | i, like, things, fascinating |
Step 2. Fix vocabulary and count repeats
| Document | fascinating | i | language | like | love | natural | processing | study | things |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 |
| 2 | 0 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 0 |
| 3 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 |
Step 3. Subtract coordinates
[1, -1, -1, -1, -1, -1, 0, -1, 0]
Step 4. Square, sum and take the root
1 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 0 = 7; √7 = 2.645751311
Final answer
Euclidean distance=2.645751311. Natural and Language each occur twice in sentence2; preserve both counts.