Deep Learning · TEE · formulas

ReLU and its derivative

ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side.

f(z)=\max(0,z)LaTeX: f(z)=\max(0,z)
f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases}LaTeX: f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases}

Drill 8 Activation contrast

Question

Question. For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.

Answer — step by step

Given

Inputs z = [−3,0,2].

Leaky-ReLU negative slope α = 0.1.

For the sigmoid derivative, use z = 0.

To find

Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.

Formula — which one and why

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

\operatorname{ReLU}(z)=\max(0,z)LaTeX: \operatorname{ReLU}(z)=\max(0,z)
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

p=\frac{1}{1+e^{-z}}LaTeX: p=\frac{1}{1+e^{-z}}
\frac{dp}{dz}=p(1-p)LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Solution — step by step

Formulas and their meanings

ReLU compares the input with zero and returns the larger value.

\operatorname{ReLU}(z)=\max(0,z)LaTeX: \operatorname{ReLU}(z)=\max(0,z)

Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.

\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Step 1: apply ReLU separately to all three inputs

\max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2LaTeX: \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2

ReLU output = [0, 0, 2].

Step 2: use α = 0.1 only on the negative input

0.1\times(-3)=-0.3LaTeX: 0.1\times(-3)=-0.3

The zero and positive entries remain 0 and 2. Leaky ReLU output = [−0.3, 0, 2].

Step 3: calculate sigmoid before its derivative

\sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]LaTeX: \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]
\sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5LaTeX: \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5
\sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25LaTeX: \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25

The sigmoid value is 0.5 but its slope is 0.25. They are different quantities.

ReLU example 1

Question. For preactivation z = -2, calculate the ReLU output and state its derivative.

Given

z = -2; activation = ReLU.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

a=\max(0,z)LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

-2 is negative, so use the negative branch.

Step 2: calculate the output.

a=0LaTeX: a=0

Step 3: state the slope on that branch.

f\prime(-2)=0LaTeX: f\prime(-2)=0

Final answer: a ≈ 0.000000000. Derivative ≈ 0.000000000.

ReLU example 2

Question. For preactivation z = 0, calculate the ReLU output and state its derivative.

Given

z = 0; activation = ReLU.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

a=\max(0,z)LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

Both branches meet at zero. The value is zero, but the left and right slopes differ.

Step 2: calculate the output.

a=0LaTeX: a=0

Step 3: state the slope on that branch.

The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.

Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.

ReLU example 3

Question. For preactivation z = 2, calculate the ReLU output and state its derivative.

Given

z = 2; activation = ReLU.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

a=\max(0,z)LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

2 is positive, so retain it.

Step 2: calculate the output.

a=2LaTeX: a=2

Step 3: state the slope on that branch.

f\prime(2)=1LaTeX: f\prime(2)=1

Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.