Deep Learning · MT2 · section

Drill 8 Activation contrast

MT2 numerical practice from direct to mixed

Question

Question. For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.

Answer — step by step

Given

Inputs z = [−3,0,2].

Leaky-ReLU negative slope α = 0.1.

For the sigmoid derivative, use z = 0.

To find

Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.

Formula — which one and why

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

\operatorname{ReLU}(z)=\max(0,z)LaTeX: \operatorname{ReLU}(z)=\max(0,z)
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

p=\frac{1}{1+e^{-z}}LaTeX: p=\frac{1}{1+e^{-z}}
\frac{dp}{dz}=p(1-p)LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Solution — step by step

Formulas and their meanings

ReLU compares the input with zero and returns the larger value.

\operatorname{ReLU}(z)=\max(0,z)LaTeX: \operatorname{ReLU}(z)=\max(0,z)

Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.

\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Step 1: apply ReLU separately to all three inputs

\max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2LaTeX: \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2

ReLU output = [0, 0, 2].

Step 2: use α = 0.1 only on the negative input

0.1\times(-3)=-0.3LaTeX: 0.1\times(-3)=-0.3

The zero and positive entries remain 0 and 2. Leaky ReLU output = [−0.3, 0, 2].

Step 3: calculate sigmoid before its derivative

\sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]LaTeX: \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]
\sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5LaTeX: \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5
\sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25LaTeX: \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25

The sigmoid value is 0.5 but its slope is 0.25. They are different quantities.