F23 Q2b

Answer — step by step

An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.

Sigmoid

LaTeX: \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1

Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.

Hyperbolic tangent

LaTeX: \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1

The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.

ReLU

LaTeX: \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases}

The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.

Leaky ReLU

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1

Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.

Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.

Facial recognition: a CNN is a suitable DL choice because the input contains spatial visual patterns. It learns local and hierarchical features with shared filters, reducing dependence on manually selected face descriptors. One channel means grayscale. With limited target data, pretrained features and carefully chosen augmentation are preferable starting points; the image dimensions alone do not guarantee enough data for training from scratch.