Deep Learning · TEE · formulas

Stable softmax

Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic.

p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}
c=\max_j z_jLaTeX: c=\max_j z_j

Drill 9 Softmax and true class loss

Question

Question. Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy.

Answer — step by step

Given

Logits = [1,2,0].

One-hot target = [0,1,0]: class 2 is correct.

To find

Find all three softmax probabilities and CCE.

Formula — which one and why

Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.

M=\max_j z_jLaTeX: M=\max_j z_j
p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}LaTeX: p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}

Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.

Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.

yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.

L=-\sum_j y_j\ln p_jLaTeX: L=-\sum_j y_j\ln p_j
L=-\ln p_cLaTeX: L=-\ln p_c

Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.

Solution — step by step

Meaning and formulas

The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.

p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_jLaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j

Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.

L_{\mathrm{CCE}}=-\sum_i y_i\ln p_iLaTeX: L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i

Step 1: subtract the largest score

The largest of [1,2,0] is 2.

[1-2,\;2-2,\;0-2]=[-1,0,-2]LaTeX: [1-2,\;2-2,\;0-2]=[-1,0,-2]

Step 2: calculate the three exponentials

e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283LaTeX: e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283

Step 3: add them to get one common denominator

D=0.367879441+1+0.135335283=1.503214724LaTeX: D=0.367879441+1+0.135335283=1.503214724

Step 4: divide each exponential by this denominator

p_1=\frac{0.367879441}{1.503214724}=0.244728471LaTeX: p_1=\frac{0.367879441}{1.503214724}=0.244728471
p_2=\frac{1}{1.503214724}=0.665240956LaTeX: p_2=\frac{1}{1.503214724}=0.665240956
p_3=\frac{0.135335283}{1.503214724}=0.090030573LaTeX: p_3=\frac{0.135335283}{1.503214724}=0.090030573

The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.

Step 5: use the probability of the correct class in CCE

L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2LaTeX: L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2
L=-\ln(0.665240956)=0.407605964LaTeX: L=-\ln(0.665240956)=0.407605964

Final answer: probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.

Softmax for [0, 0, 0]

Question. Find stable softmax probabilities for logits [0, 0, 0].

Given

Logits = [0, 0, 0]; there are three mutually exclusive classes.

To find

All three class probabilities.

Formula — which one and why

Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.

A logit is a raw class score. c is the largest score; pᵢ is probability for class i.

c=\max_j z_jLaTeX: c=\max_j z_j
p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}

Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.

Solution — step by step

Step 1: subtract the largest score from every entry.

z-c=\begin{bmatrix}0&0&0\end{bmatrix}LaTeX: z-c=\begin{bmatrix}0&0&0\end{bmatrix}

Step 2: exponentiate shifted entry 1.

e^{0}\approx1.000000000LaTeX: e^{0}\approx1.000000000

Step 3: exponentiate shifted entry 2.

e^{0}\approx1.000000000LaTeX: e^{0}\approx1.000000000

Step 4: exponentiate shifted entry 3.

e^{0}\approx1.000000000LaTeX: e^{0}\approx1.000000000

Step 5: add all exponentials into one denominator.

D=1.000000000+1.000000000+1.000000000\approx3.000000000LaTeX: D=1.000000000+1.000000000+1.000000000\approx3.000000000

Step 6: divide entry 1 by the same denominator.

p_1=\frac{1.000000000}{3.000000000}\approx0.333333333LaTeX: p_1=\frac{1.000000000}{3.000000000}\approx0.333333333

Step 7: divide entry 2 by the same denominator.

p_2=\frac{1.000000000}{3.000000000}\approx0.333333333LaTeX: p_2=\frac{1.000000000}{3.000000000}\approx0.333333333

Step 8: divide entry 3 by the same denominator.

p_3=\frac{1.000000000}{3.000000000}\approx0.333333333LaTeX: p_3=\frac{1.000000000}{3.000000000}\approx0.333333333

Final answer: Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding.

Softmax for [1, 2, 3]

Question. Find stable softmax probabilities for logits [1, 2, 3].

Given

Logits = [1, 2, 3]; there are three mutually exclusive classes.

To find

All three class probabilities.

Formula — which one and why

Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.

A logit is a raw class score. c is the largest score; pᵢ is probability for class i.

c=\max_j z_jLaTeX: c=\max_j z_j
p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}

Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.

Solution — step by step

Step 1: subtract the largest score from every entry.

z-c=\begin{bmatrix}-2&-1&0\end{bmatrix}LaTeX: z-c=\begin{bmatrix}-2&-1&0\end{bmatrix}

Step 2: exponentiate shifted entry 1.

e^{-2}\approx0.135335283LaTeX: e^{-2}\approx0.135335283

Step 3: exponentiate shifted entry 2.

e^{-1}\approx0.367879441LaTeX: e^{-1}\approx0.367879441

Step 4: exponentiate shifted entry 3.

e^{0}\approx1.000000000LaTeX: e^{0}\approx1.000000000

Step 5: add all exponentials into one denominator.

D=0.135335283+0.367879441+1.000000000\approx1.503214724LaTeX: D=0.135335283+0.367879441+1.000000000\approx1.503214724

Step 6: divide entry 1 by the same denominator.

p_1=\frac{0.135335283}{1.503214724}\approx0.090030573LaTeX: p_1=\frac{0.135335283}{1.503214724}\approx0.090030573

Step 7: divide entry 2 by the same denominator.

p_2=\frac{0.367879441}{1.503214724}\approx0.244728471LaTeX: p_2=\frac{0.367879441}{1.503214724}\approx0.244728471

Step 8: divide entry 3 by the same denominator.

p_3=\frac{1.000000000}{1.503214724}\approx0.665240956LaTeX: p_3=\frac{1.000000000}{1.503214724}\approx0.665240956

Final answer: Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.