Deep Learning · MT2 · section

MT1B Q3

Prerequisite repair and teacher backprop examples

Question

Question (3 marks). Calculate sigmoid, ReLU and tanh for z = [−2,0,2].

Answer — step by step

Given

Input scores = [−2,0,2].

Required activations: sigmoid, ReLU and tanh.

To find

Calculate all three activation outputs for each score.

Formula — which one and why

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

$p=\frac{1}{1+e^{-z}}$ LaTeX: p=\frac{1}{1+e^{-z}}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

$\operatorname{ReLU}(z)=\max(0,z)$ LaTeX: \operatorname{ReLU}(z)=\max(0,z)

$\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}$ LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Tanh and its slope. Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².

z = value entering tanh; a = its output. A prime means derivative, or local rate of change.