# Drill 8 Activation contrast

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s8

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s8/

## Question

**Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.

## Answer — step by step

### Given

Inputs z = [−3,0,2].

Leaky-ReLU negative slope α = 0.1.

For the sigmoid derivative, use z = 0.

### To find

Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.

### Formula — which one and why

**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$





$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

### Solution — step by step

### Formulas and their meanings

ReLU compares the input with zero and returns the larger value.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$



Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.



$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



### Step 1: apply ReLU separately to all three inputs



$$
\max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2
$$



**ReLU output = [0, 0, 2].**

### Step 2: use α = 0.1 only on the negative input



$$
0.1\times(-3)=-0.3
$$



The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].**

### Step 3: calculate sigmoid before its derivative



$$
\sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]
$$





$$
\sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5
$$





$$
\sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25
$$



The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities.