# Leaky ReLU

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/11

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/11/

Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function.



$$
f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases}
$$



## Drill 8 Activation contrast

## Question

**Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.

## Answer — step by step

### Given

Inputs z = [−3,0,2].

Leaky-ReLU negative slope α = 0.1.

For the sigmoid derivative, use z = 0.

### To find

Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.

### Formula — which one and why

**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$





$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

### Solution — step by step

### Formulas and their meanings

ReLU compares the input with zero and returns the larger value.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$



Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.



$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



### Step 1: apply ReLU separately to all three inputs



$$
\max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2
$$



**ReLU output = [0, 0, 2].**

### Step 2: use α = 0.1 only on the negative input



$$
0.1\times(-3)=-0.3
$$



The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].**

### Step 3: calculate sigmoid before its derivative



$$
\sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]
$$





$$
\sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5
$$





$$
\sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25
$$



The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities.

### Leaky ReLU example 1

**Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.**

### Given

z = -2; activation = Leaky ReLU.

α = 0.1.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

-2 is negative, so use the negative branch.

**Step 2: calculate the output.**



$$
a=-0.2
$$



**Step 3: state the slope on that branch.**



$$
f\prime(-2)=0.1
$$



**Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000.

### Leaky ReLU example 2

**Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.**

### Given

z = 0; activation = Leaky ReLU.

α = 0.1.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

Both branches meet at zero. The value is zero, but the left and right slopes differ.

**Step 2: calculate the output.**



$$
a=0
$$



**Step 3: state the slope on that branch.**

The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.

**Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero.

### Leaky ReLU example 3

**Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.**

### Given

z = 2; activation = Leaky ReLU.

α = 0.1.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

2 is positive, so retain it.

**Step 2: calculate the output.**



$$
a=2
$$



**Step 3: state the slope on that branch.**



$$
f\prime(2)=1
$$



**Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000.