# R23 Q2b

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s6

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s6/

## Question

**Question (10 marks).** Justify ReLU over sigmoid/tanh for deep hidden layers. For predicting brand ratings on a scale of 10, choose output activation and loss. Diagnose 2% training versus 18% validation error and suggest remedies.

## Answer — step by step

An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.

**Sigmoid**



$$
\sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1
$$



Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.

**Hyperbolic tangent**



$$
\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1
$$



The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.

**ReLU**



$$
\operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases}
$$



The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.

**Leaky ReLU**



$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1
$$



Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.

Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.

**Rating regression:**  for a continuous score, one linear output with MSE is a standard baseline. If strict 0–10 bounds are required, use 10 × sigmoid(z) and the matching regression loss, noting saturation near the endpoints. If labels are categorical rating bins, a categorical formulation would be different; state the continuous-score assumption.

**2% versus 18%:**  if errors use the same comparable metric and distribution, this large generalization gap strongly suggests overfitting. Use appropriate regularization, early stopping, more representative data and possibly a smaller model; inspect leakage and preprocessing too.

Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.



$$
J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2
$$



L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.

Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.