Deep Learning · TEE · section

Drill 48 Mixed dropout and a momentum update

Numerical practice from direct to mixed

Question

Question. Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.

Answer — step by step

Given

Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.

Linear-output weights = [0.2,0.3]; bias = 0; target = 1.

Half squared loss; update only the output weights.

Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.

To find

Find dropped/scaled activations, prediction, gradients, new velocities and new output weights.

Formula — which one and why

Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.

a_i^{\mathrm{train}}=\frac{m_ia_i}{q}LaTeX: a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
a_i^{\mathrm{test}}=a_iLaTeX: a_i^{\mathrm{test}}=a_i

Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.

Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.

v_t=\beta v_{t-1}-\eta g_tLaTeX: v_t=\beta v_{t-1}-\eta g_t
\theta_t=\theta_{t-1}+v_tLaTeX: \theta_t=\theta_{t-1}+v_t

Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.

Solution — step by step

Formulas used

Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.

a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_jLaTeX: a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j

The signed-velocity momentum convention is:

v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}}LaTeX: v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}}

Step 1: apply inverted dropout

a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0LaTeX: a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0

Step 2: prediction and error

o=0.2(2)+0.3(0)+0=0.4LaTeX: o=0.2(2)+0.3(0)+0=0.4
o-y=0.4-1=-0.6LaTeX: o-y=0.4-1=-0.6

Step 3: current weight gradients

g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0LaTeX: g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0

Step 4: remembered movement plus new correction

v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21LaTeX: v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21
v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09LaTeX: v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09

Step 5: add velocities to old weights

w_{1,\mathrm{new}}=0.2+0.21=0.41LaTeX: w_{1,\mathrm{new}}=0.2+0.21=0.41
w_{2,\mathrm{new}}=0.3-0.09=0.21LaTeX: w_{2,\mathrm{new}}=0.3-0.09=0.21

Final answer: [0.41, 0.21]. The second weight moves even with zero current gradient because its old momentum is nonzero.