# R26 Q6b

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s13

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s13/

## Question

**Question (10 marks).** For a 10-layer network using sigmoid: compute f(6), compute its derivative at 6, explain its effect on backpropagation, and calculate softmax for [2.3,4.5,3.2].

## Answer — step by step

### Given

Sigmoid is evaluated at z = 6 in a ten-layer network.

Softmax scores = [2.3,4.5,3.2].

### To find

Calculate sigmoid and its derivative at 6, explain the gradient effect and calculate all three softmax probabilities.

### Formula — which one and why

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.



$$
M=\max_j z_j
$$





$$
p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}
$$



**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.

### Solution — step by step

f(6) =  $\frac{1}{1+e^{-6}}$  = **0.997527377**. Derivative = f(6)[1−f(6)] = 0.997527377 × 0.002472623 = **0.002466509**. This is a saturated sigmoid with a very small slope. Repeatedly multiplying small activation derivatives and weight factors can weaken earlier-layer gradients; the exact gradient also depends on weights and all preactivations, so it is not simply this derivative raised to 10 unless those conditions are assumed.

**Step 1 stabilize:**  subtract the largest logit 4.5 from every value: [-2.2, 0, -1.3]. This does not change softmax.

**Step 2 exponentiate:**  exp values = [0.110803158, 1, 0.272531793].

**Step 3 denominator:**  sum = 1.383334951.

**Step 4 divide each value by this same sum:**  probabilities = **[0.080098575, 0.722890721, 0.197010704]**. **Check:**  all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.