Deep Learning · TEE · section

Drill 9 Softmax and true class loss

Numerical practice from direct to mixed

Question

Question. Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy.

Answer — step by step

Given

Logits = [1,2,0].

One-hot target = [0,1,0]: class 2 is correct.

To find

Find all three softmax probabilities and CCE.

Formula — which one and why

Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.

M=\max_j z_jLaTeX: M=\max_j z_j
p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}LaTeX: p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}

Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.

Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.

yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.

L=-\sum_j y_j\ln p_jLaTeX: L=-\sum_j y_j\ln p_j
L=-\ln p_cLaTeX: L=-\ln p_c

Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.

Solution — step by step

Meaning and formulas

The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.

p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_jLaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j

Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.

L_{\mathrm{CCE}}=-\sum_i y_i\ln p_iLaTeX: L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i

Step 1: subtract the largest score

The largest of [1,2,0] is 2.

[1-2,\;2-2,\;0-2]=[-1,0,-2]LaTeX: [1-2,\;2-2,\;0-2]=[-1,0,-2]

Step 2: calculate the three exponentials

e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283LaTeX: e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283

Step 3: add them to get one common denominator

D=0.367879441+1+0.135335283=1.503214724LaTeX: D=0.367879441+1+0.135335283=1.503214724

Step 4: divide each exponential by this denominator

p_1=\frac{0.367879441}{1.503214724}=0.244728471LaTeX: p_1=\frac{0.367879441}{1.503214724}=0.244728471
p_2=\frac{1}{1.503214724}=0.665240956LaTeX: p_2=\frac{1}{1.503214724}=0.665240956
p_3=\frac{0.135335283}{1.503214724}=0.090030573LaTeX: p_3=\frac{0.135335283}{1.503214724}=0.090030573

The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.

Step 5: use the probability of the correct class in CCE

L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2LaTeX: L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2
L=-\ln(0.665240956)=0.407605964LaTeX: L=-\ln(0.665240956)=0.407605964

Final answer: probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.