F25 Q6b

Answer — step by step

An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.

Sigmoid

LaTeX: \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1

Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.

Hyperbolic tangent

LaTeX: \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1

The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.

ReLU

LaTeX: \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases}

The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.

Leaky ReLU

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1

Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.

Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.