Natural Language Processing · TEE · papers

Q6a · GloVe

2022-11-22 · final · recovered AI paper

Previous examination · complete worked answers

Question. Explain GloVe word embeddings. [10 marks]

Answer

GloVe (Global Vectors) learns word embeddings by fitting weighted relationships in a global word–context co-occurrence matrix. Word2Vec uses local prediction examples; GloVe uses aggregate counts directly in its objective.

X_i=\sum_k X_{ik},\qquad P_{ik}=\frac{X_{ik}}{X_i}.

$X_{ik}$ is how often context word $k$ appears around target $i$ . $X_i$ is the total of row $i$ . $P_{ik}$ is the resulting context probability.

The ratio $P(k\mid i)/P(k\mid j)$ helps identify how a probe context differs between two words. For targets ice and steam, a probe such as solid should favour ice; gas should favour steam; a probe shared similarly by both gives a ratio near one. These are data tendencies, not universal numerical identities.

Question. For context solid, ice has count 12 out of row total 20; steam has count 2 out of row total 40. Compute P(solid|ice), P(solid|steam) and their ratio.

Given

X(ice,solid)=12, X(ice)=20; X(steam,solid)=2, X(steam)=40.

To find

Two conditional probabilities and the ice/steam ratio.

Formula and why it applies

$P(k\mid i)=X_{ik}/X_i$ , and $R=P(k\mid ice)/P(k\mid steam)$ .

Step 1. Ice probability

$12/20=0.6$ .

Step 2. Steam probability

$2/40=0.05$ .

Step 3. Ratio

$0.6/0.05=12$ .

Answer. The probe solid is 12 times as probable in ice's context under these counts.

Question. For probe water, ice and steam both have row total 50. Counts are 10 and 10. Calculate the ratio.

Given

Both probabilities use a denominator of 50.

To find

The probability ratio.

Formula and why it applies

Use row-normalized counts before taking the ratio.

Step 1. Ice probability

$10/50=0.2$ .

Step 2. Steam probability

$10/50=0.2$ .

Step 3. Divide

$0.2/0.2=1$ .

Answer. Ratio 1: this probe does not distinguish the targets in this toy data.

Question. For probe gas, counts are 1/100 around ice and 20/100 around steam. Compute the ice-to-steam ratio.

Given

Conditional counts have the same row denominator.

To find

The ratio and interpretation.

Formula and why it applies

$R=P(gas\mid ice)/P(gas\mid steam)$ .

Step 1. Ice probability

$1/100=0.01$ .

Step 2. Steam probability

$20/100=0.2$ .

Step 3. Ratio

$0.01/0.2=0.05$ .

Answer. Ratio 0.05, so gas is much more strongly associated with steam.

Ratio >1: pehla word favoured. <1: doosra. ≈1: probe dono mein similar.