# R24 Q3a

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s6

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s6/

## Question

**Question (10 marks).** Differentiate sigmoid and ReLU(4). Explain why sigmoid/tanh are often avoided in deep hidden layers(3). Compute softmax of [2.1,4.8,3.5](3).

## Answer — step by step

### Given

Softmax scores = [2.1,4.8,3.5].

The numerals (4),(3),(3) in the prompt are marks allocations, not activation inputs.

### To find

Differentiate sigmoid/ReLU, explain hidden-layer saturation and calculate softmax.

### Formula — which one and why

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.



$$
M=\max_j z_j
$$





$$
p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}
$$



**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.

**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$





$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

### Solution — step by step

An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.

**Sigmoid**



$$
\sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1
$$



Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.

**Hyperbolic tangent**



$$
\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1
$$



The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.

**ReLU**



$$
\operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases}
$$



The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.

**Leaky ReLU**



$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1
$$



Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.

Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.

**Step 1 stabilize:**  subtract the largest logit 4.8 from every value: [-2.7, 0, -1.3]. This does not change softmax.

**Step 2 exponentiate:**  exp values = [0.067205513, 1, 0.272531793].

**Step 3 denominator:**  sum = 1.339737306.

**Step 4 divide each value by this same sum:**  probabilities = **[0.050163202, 0.746414984, 0.203421814]**. **Check:**  all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.