Deep Learning · TEE · formulas

Inverted dropout

Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated.

q=1-pLaTeX: q=1-p
a_{train}=\frac{m\odot a}{q}LaTeX: a_{train}=\frac{m\odot a}{q}
a_{test}=aLaTeX: a_{test}=a

Model answer: dropout with training and inference explained

Question

Question. Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling.

Answer — step by step

Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights.

The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge.

Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop.

Inverted dropout during training

a'=\frac{ma}{q}LaTeX: a'=\frac{ma}{q}

A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks.

For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is:

0.5(6)+0.5(0)=3LaTeX: 0.5(6)+0.5(0)=3

Inference

Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions.

Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data.

Dropout removes activations during training

Definition. Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training.

Explanation and working

Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count.

Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge.

Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions.

a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout}LaTeX: a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout}
a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout}LaTeX: a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout}

Question. Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs.

Given

Activations = [0.8,−0.4,0.6,0.2].

Dropout rate = 50%, so keep probability q = 0.5.

Chosen mask = [1,0,1,0].

To find

Find training outputs under inverted dropout.

Formula — which one and why

Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.

a_i^{\mathrm{train}}=\frac{m_ia_i}{q}LaTeX: a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
a_i^{\mathrm{test}}=a_iLaTeX: a_i^{\mathrm{test}}=a_i

Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.

Solution — step by step

Step 1. q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key.

Step 2. For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = [1.6,0,1.2,0]. Inference then uses the full network without that extra scaling. State the convention in an answer.

Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition.

Drill 33 Inverted dropout

Question

Question. Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference.

Answer — step by step

Given

Activations = [2,−1,0,3].

Keep probability q = 0.75; keep mask = [1,0,1,1].

To find

Find training outputs and state the inference rule.

Formula — which one and why

Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.

a_i^{\mathrm{train}}=\frac{m_ia_i}{q}LaTeX: a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
a_i^{\mathrm{test}}=a_iLaTeX: a_i^{\mathrm{test}}=a_i

Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.

Solution — step by step

Meaning and formula

The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved.

a'_j=\frac{m_j a_j}{q}LaTeX: a'_j=\frac{m_j a_j}{q}

The multiplication is entry by entry; there is no dot-product sum.

Step 1: apply the mask

[1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3]LaTeX: [1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3]

Step 2: divide each result by 0.75

a'_1=\frac2{0.75}=2.666666667LaTeX: a'_1=\frac2{0.75}=2.666666667
a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0LaTeX: a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0
a'_4=\frac3{0.75}=4LaTeX: a'_4=\frac3{0.75}=4

Training output = [2.666666667, 0, 0, 4]. A kept activation that was already zero stays zero.

During inference

Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training.

Drill 48 Mixed dropout and a momentum update

Question

Question. Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.

Answer — step by step

Given

Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.

Linear-output weights = [0.2,0.3]; bias = 0; target = 1.

Half squared loss; update only the output weights.

Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.

To find

Find dropped/scaled activations, prediction, gradients, new velocities and new output weights.

Formula — which one and why

Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.

a_i^{\mathrm{train}}=\frac{m_ia_i}{q}LaTeX: a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
a_i^{\mathrm{test}}=a_iLaTeX: a_i^{\mathrm{test}}=a_i

Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.

Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.

v_t=\beta v_{t-1}-\eta g_tLaTeX: v_t=\beta v_{t-1}-\eta g_t
\theta_t=\theta_{t-1}+v_tLaTeX: \theta_t=\theta_{t-1}+v_t

Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.

Solution — step by step

Formulas used

Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.

a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_jLaTeX: a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j

The signed-velocity momentum convention is:

v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}}LaTeX: v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}}

Step 1: apply inverted dropout

a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0LaTeX: a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0

Step 2: prediction and error

o=0.2(2)+0.3(0)+0=0.4LaTeX: o=0.2(2)+0.3(0)+0=0.4
o-y=0.4-1=-0.6LaTeX: o-y=0.4-1=-0.6

Step 3: current weight gradients

g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0LaTeX: g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0

Step 4: remembered movement plus new correction

v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21LaTeX: v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21
v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09LaTeX: v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09

Step 5: add velocities to old weights

w_{1,\mathrm{new}}=0.2+0.21=0.41LaTeX: w_{1,\mathrm{new}}=0.2+0.21=0.41
w_{2,\mathrm{new}}=0.3-0.09=0.21LaTeX: w_{2,\mathrm{new}}=0.3-0.09=0.21

Final answer: [0.41, 0.21]. The second weight moves even with zero current gradient because its old momentum is nonzero.