Deep Learning · MT2 · section

Dropout removes activations during training

Regularization and generalization

Definition. Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training.

Explanation and working

Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count.

Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge.

Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions.

a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout}LaTeX: a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout}
a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout}LaTeX: a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout}

Question. Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs.

Given

Activations = [0.8,−0.4,0.6,0.2].

Dropout rate = 50%, so keep probability q = 0.5.

Chosen mask = [1,0,1,0].

To find

Find training outputs under inverted dropout.

Formula — which one and why

Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.

a_i^{\mathrm{train}}=\frac{m_ia_i}{q}LaTeX: a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
a_i^{\mathrm{test}}=a_iLaTeX: a_i^{\mathrm{test}}=a_i

Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.

Solution — step by step

Step 1. q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key.

Step 2. For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = [1.6,0,1.2,0]. Inference then uses the full network without that extra scaling. State the convention in an answer.

Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition.