# Word analogies and sentence embeddings

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u5-sentence

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u5-sentence/

Formulas explained through full worked examples

**A sentence embedding** represents an entire sentence by one fixed-length vector. A basic method averages word embeddings. More advanced encoders learn a sentence representation from context and task objectives.



$$
v_{\text{sentence}}=\frac1n\sum_{i=1}^{n}v_{w_i}.
$$



This means add corresponding coordinates and divide each coordinate by the number of words. Averaging ignores order: “dog bites man” and “man bites dog” have the same average if their words are identical.

**Question. Average the word vectors [1,0], [0,2] and [2,1] to represent a three-word sentence.**

#### Given

Three two-dimensional vectors with equal weights.

#### To find

The two-dimensional sentence vector.

#### Formula and why it applies

Use coordinate-wise mean 

$$
v=(v_1+v_2+v_3)/3
$$

.

#### Step 1. First coordinate



$$
1+0+2=3
$$

; 

$$
3/3=1
$$

.

#### Step 2. Second coordinate



$$
0+2+1=3
$$

; 

$$
3/3=1
$$

.

**Answer.** Sentence vector = [1,1].

**Question. Toy word vectors are king=[3,2], man=[1,0], woman=[1,1]. Find king−man+woman and compare to queen=[3,3].**

#### Given

All vectors use the same two-coordinate space.

#### To find

The analogy vector and its distance from queen.

#### Formula and why it applies

Vector arithmetic is coordinate by coordinate. Euclidean distance is the square root of the sum of squared coordinate differences.

#### Step 1. Subtract man



$$
[3-1,2-0]=[2,2]
$$

.

#### Step 2. Add woman



$$
[2+1,2+1]=[3,3]
$$

.

#### Step 3. Compare with queen



$$
\sqrt{(3-3)^2+(3-3)^2}=0
$$

.

**Answer.** The toy analogy exactly equals queen. Real learned embeddings only approximate some analogies.

**Contextual encoders** can assign “bank” different representations in different sentences. BERT and RoBERTa produce contextual token representations; obtaining strong sentence similarity vectors requires an appropriate pooling/training method, not simply assuming every raw token vector is a sentence embedding. Universal Sentence Encoder is designed to encode sentence-level meaning. Skip-thought predicts surrounding sentences; InferSent learns from natural-language-inference supervision.

Sentence average: har column add karo, words ki count se divide. Word order ka loss yaad rakh.