# MT1B Q3

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s8

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s8/

## Question

**Question (3 marks).** Calculate sigmoid, ReLU and tanh for z = [−2,0,2].

## Answer — step by step

### Given

Input scores = [−2,0,2].

Required activations: sigmoid, ReLU and tanh.

### To find

Calculate all three activation outputs for each score.

### Formula — which one and why

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$





$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

**Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².

z = value entering tanh; a = its output. A prime means derivative, or local rate of change.



$$
a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}}
$$





$$
\frac{da}{dz}=1-a^2
$$



**Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square.

### Solution — step by step

### Preactivation z = -2

**Step 1: negate z for sigmoid.**



$$
-z=2
$$



**Step 2: evaluate the exponential.**



$$
e^{-z}\approx7.389056099
$$



**Step 3: add one to form the denominator.**



$$
1+7.389056099=8.389056099
$$



**Step 4: take the reciprocal for sigmoid.**



$$
\sigma(-2)=\frac1{8.389056099}\approx0.119202922
$$



**Step 5: compare z with zero for ReLU.**



$$
\mathrm{ReLU}(-2)=\max(0,-2)=0
$$



**Step 6: calculate the tanh numerator.**



$$
e^{-2}-e^{2}\approx0.135335283-7.389056099=-7.253720816
$$



**Step 7: calculate the tanh denominator.**



$$
e^{-2}+e^{2}\approx0.135335283+7.389056099=7.524391382
$$



**Step 8: divide for tanh.**



$$
\tanh(-2)\approx\frac{-7.253720816}{7.524391382}\approx-0.964027580
$$



### Preactivation z = 0

**Step 1: negate z for sigmoid.**



$$
-z=0
$$



**Step 2: evaluate the exponential.**



$$
e^{-z}\approx1.000000000
$$



**Step 3: add one to form the denominator.**



$$
1+1.000000000=2.000000000
$$



**Step 4: take the reciprocal for sigmoid.**



$$
\sigma(0)=\frac1{2.000000000}\approx0.500000000
$$



**Step 5: compare z with zero for ReLU.**



$$
\mathrm{ReLU}(0)=\max(0,0)=0
$$



**Step 6: calculate the tanh numerator.**



$$
e^{0}-e^{0}\approx1.000000000-1.000000000=0.000000000
$$



**Step 7: calculate the tanh denominator.**



$$
e^{0}+e^{0}\approx1.000000000+1.000000000=2.000000000
$$



**Step 8: divide for tanh.**



$$
\tanh(0)\approx\frac{0.000000000}{2.000000000}\approx0.000000000
$$



### Preactivation z = 2

**Step 1: negate z for sigmoid.**



$$
-z=-2
$$



**Step 2: evaluate the exponential.**



$$
e^{-z}\approx0.135335283
$$



**Step 3: add one to form the denominator.**



$$
1+0.135335283=1.135335283
$$



**Step 4: take the reciprocal for sigmoid.**



$$
\sigma(2)=\frac1{1.135335283}\approx0.880797078
$$



**Step 5: compare z with zero for ReLU.**



$$
\mathrm{ReLU}(2)=\max(0,2)=2
$$



**Step 6: calculate the tanh numerator.**



$$
e^{2}-e^{-2}\approx7.389056099-0.135335283=7.253720816
$$



**Step 7: calculate the tanh denominator.**



$$
e^{2}+e^{-2}\approx7.389056099+0.135335283=7.524391382
$$



**Step 8: divide for tanh.**



$$
\tanh(2)\approx\frac{7.253720816}{7.524391382}\approx0.964027580
$$



**Final answer:** sigmoid = [0.119202922, 0.5, 0.880797078]; ReLU = [0, 0, 2]; tanh = [−0.964027580, 0, 0.964027580].