Natural Language Processing · TEE · formulas

Word analogies and sentence embeddings

Representation & similarity

Formulas explained through full worked examples

A sentence embedding represents an entire sentence by one fixed-length vector. A basic method averages word embeddings. More advanced encoders learn a sentence representation from context and task objectives.

v_{\text{sentence}}=\frac1n\sum_{i=1}^{n}v_{w_i}.

This means add corresponding coordinates and divide each coordinate by the number of words. Averaging ignores order: “dog bites man” and “man bites dog” have the same average if their words are identical.

Question. Average the word vectors [1,0], [0,2] and [2,1] to represent a three-word sentence.

Given

Three two-dimensional vectors with equal weights.

To find

The two-dimensional sentence vector.

Formula and why it applies

Use coordinate-wise mean $v=(v_1+v_2+v_3)/3$ .

Step 1. First coordinate

$1+0+2=3$ ; $3/3=1$ .

Step 2. Second coordinate

$0+2+1=3$ ; $3/3=1$ .

Answer. Sentence vector = [1,1].

Question. Toy word vectors are king=[3,2], man=[1,0], woman=[1,1]. Find king−man+woman and compare to queen=[3,3].

Given

All vectors use the same two-coordinate space.

To find

The analogy vector and its distance from queen.

Formula and why it applies

Vector arithmetic is coordinate by coordinate. Euclidean distance is the square root of the sum of squared coordinate differences.

Step 1. Subtract man

$[3-1,2-0]=[2,2]$ .

Step 2. Add woman

$[2+1,2+1]=[3,3]$ .

Step 3. Compare with queen

$\sqrt{(3-3)^2+(3-3)^2}=0$ .

Answer. The toy analogy exactly equals queen. Real learned embeddings only approximate some analogies.

Contextual encoders can assign “bank” different representations in different sentences. BERT and RoBERTa produce contextual token representations; obtaining strong sentence similarity vectors requires an appropriate pooling/training method, not simply assuming every raw token vector is a sentence embedding. Universal Sentence Encoder is designed to encode sentence-level meaning. Skip-thought predicts surrounding sentences; InferSent learns from natural-language-inference supervision.

Sentence average: har column add karo, words ki count se divide. Word order ka loss yaad rakh.