# Drill 9 Softmax and true class loss

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s9

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s9/

## Question

**Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy.

## Answer — step by step

### Given

Logits = [1,2,0].

One-hot target = [0,1,0]: class 2 is correct.

### To find

Find all three softmax probabilities and CCE.

### Formula — which one and why

**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.



$$
M=\max_j z_j
$$





$$
p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}
$$



**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.

**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.

yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.



$$
L=-\sum_j y_j\ln p_j
$$





$$
L=-\ln p_c
$$



**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.

### Solution — step by step

### Meaning and formulas

The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.



$$
p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j
$$



Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.



$$
L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i
$$



### Step 1: subtract the largest score

The largest of [1,2,0] is 2.



$$
[1-2,\;2-2,\;0-2]=[-1,0,-2]
$$



### Step 2: calculate the three exponentials



$$
e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283
$$



### Step 3: add them to get one common denominator



$$
D=0.367879441+1+0.135335283=1.503214724
$$



### Step 4: divide each exponential by this denominator



$$
p_1=\frac{0.367879441}{1.503214724}=0.244728471
$$





$$
p_2=\frac{1}{1.503214724}=0.665240956
$$





$$
p_3=\frac{0.135335283}{1.503214724}=0.090030573
$$



The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.

### Step 5: use the probability of the correct class in CCE



$$
L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2
$$





$$
L=-\ln(0.665240956)=0.407605964
$$



**Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.