Deep Learning · MT2 · section

R26 Q1c

R26 selected MT2 practice

Question

Question (5 marks). Explain L1 and L2 regularization. Apply 50% dropout to outputs [0.8,−0.4,0.6,0.2] using an explicitly assumed mask.

Answer — step by step

Given

Activations = [0.8,−0.4,0.6,0.2]; dropout rate = 50%.

The worked example assumes mask [1,0,1,0] and uses inverted dropout.

No numerical weights or λ are supplied for the regularization explanation.

To find

Explain L1/L2 and calculate post-dropout activations under the stated mask.

Formula — which one and why

L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.

λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.

$J=J_{\mathrm{data}}+\lambda\sum_i|w_i|$ LaTeX: J=J_{\mathrm{data}}+\lambda\sum_i|w_i|

$g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i)$ LaTeX: g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i)

Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.

L2 penalty with a factor of one-half. This answer uses λ/2 times the sum of squared weights. Differentiating a square supplies a factor of 2, which cancels the one-half, so the added gradient is λw. Use this same convention in the explanation below.

λ = penalty strength; J_data = original data loss; wᵢ = weight i.

$J=J_{\mathrm{data}}+\frac{\lambda}{2}\sum_iw_i^2$ LaTeX: J=J_{\mathrm{data}}+\frac{\lambda}{2}\sum_iw_i^2

$\frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+\lambda w_i$ LaTeX: \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+\lambda w_i

Yaad rakhne ka cue: L2 mein half diya hai: square ka 2 cancel, gradient mein lambda × weight.

Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.

$a_i^{\mathrm{train}}=\frac{m_ia_i}{q}$ LaTeX: a_i^{\mathrm{train}}=\frac{m_ia_i}{q}

$a_i^{\mathrm{test}}=a_i$ LaTeX: a_i^{\mathrm{test}}=a_i

Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.

Solution — step by step

Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.

J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2

L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.

Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.

Numerical with one explicitly chosen mask

Step 1: convert dropout probability to keep probability.

$q=1-0.5=0.5$ LaTeX: q=1-0.5=0.5

Step 2: state a possible mask.

Choose [1,0,1,0]; 1 means keep and 0 means drop. The question allows an assumed mask; others give different realizations.

Step 3: multiply each activation by its mask entry.

Step 4: divide the kept values by q for inverted dropout.

$0.8/0.5=1.6$ LaTeX: 0.8/0.5=1.6

$0.6/0.5=1.2$ LaTeX: 0.6/0.5=1.2

Step 5: assemble the training output.

$a_{\mathrm{train}}=[1.6,0,1.2,0]$ LaTeX: a_{\mathrm{train}}=[1.6,0,1.2,0]

Final answer: simple masking gives [0.8,0,0.6,0]; inverted training dropout gives [1.6,0,1.2,0]. At inference inverted dropout is disabled.