# Stable softmax Subject: Deep Learning Exam: MT2 Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/12 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/12/ Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ $$ c=\max_j z_j $$ ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. ### Softmax for [0, 0, 0] **Question. Find stable softmax probabilities for logits [0, 0, 0].** ### Given Logits = [0, 0, 0]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}0&0&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{0}\approx1.000000000 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{0}\approx1.000000000 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=1.000000000+1.000000000+1.000000000\approx3.000000000 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding. ### Softmax for [1, 2, 3] **Question. Find stable softmax probabilities for logits [1, 2, 3].** ### Given Logits = [1, 2, 3]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}-2&-1&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{-2}\approx0.135335283 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{-1}\approx0.367879441 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=0.135335283+0.367879441+1.000000000\approx1.503214724 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{0.135335283}{1.503214724}\approx0.090030573 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{0.367879441}{1.503214724}\approx0.244728471 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{1.503214724}\approx0.665240956 $$ **Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.