# Drill 48 Mixed dropout and a momentum update

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s48

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s48/

## Question

**Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.

## Answer — step by step

### Given

Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.

Linear-output weights = [0.2,0.3]; bias = 0; target = 1.

Half squared loss; update only the output weights.

Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.

### To find

Find dropped/scaled activations, prediction, gradients, new velocities and new output weights.

### Formula — which one and why

**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.



$$
a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
$$





$$
a_i^{\mathrm{test}}=a_i
$$



**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

### Formulas used

Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.



$$
a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j
$$



The signed-velocity momentum convention is:



$$
v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}}
$$



### Step 1: apply inverted dropout



$$
a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0
$$



### Step 2: prediction and error



$$
o=0.2(2)+0.3(0)+0=0.4
$$





$$
o-y=0.4-1=-0.6
$$



### Step 3: current weight gradients



$$
g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0
$$



### Step 4: remembered movement plus new correction



$$
v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21
$$





$$
v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09
$$



### Step 5: add velocities to old weights



$$
w_{1,\mathrm{new}}=0.2+0.21=0.41
$$





$$
w_{2,\mathrm{new}}=0.3-0.09=0.21
$$



**Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero.