# Inverted dropout

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/29

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/29/

Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated.



$$
q=1-p
$$





$$
a_{train}=\frac{m\odot a}{q}
$$





$$
a_{test}=a
$$



## Model answer: dropout with training and inference explained

## Question

**Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling.

## Answer — step by step

Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights.

![The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg)

Diagram: The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3.

The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge.

Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop.

### Inverted dropout during training



$$
a'=\frac{ma}{q}
$$



A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks.

For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is:



$$
0.5(6)+0.5(0)=3
$$



### Inference

Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions.

Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data.

## Dropout removes activations during training

**Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training.

### Explanation and working

Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count.

![Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout.svg)

Diagram: Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.. Labels: Dropout temporarily removes selected activations | Dropout temporarily removes selected activations | Example: keep probability q = 0.5. Inverted dropout during training. | Original | 0.8 | -0.4 | 0.6 | 0.2 | Mask | 1 | 0 | 1 | 0 | After mask ÷ q | 1.6 | 0 | 1.2 | 0 | The kept values double: 0.8 ÷ 0.5 = 1.6 and 0.6 ÷ 0.5 = 1.2. At inference, no | mask is used.

Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge.

Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions.



$$
a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout}
$$





$$
a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout}
$$



**Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs.

### Given

Activations = [0.8,−0.4,0.6,0.2].

Dropout rate = 50%, so keep probability q = 0.5.

Chosen mask = [1,0,1,0].

### To find

Find training outputs under inverted dropout.

### Formula — which one and why

**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.



$$
a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
$$





$$
a_i^{\mathrm{test}}=a_i
$$



**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.

### Solution — step by step

**Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key.

**Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer.

Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition.

## Drill 31 Inverted dropout

## Question

**Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference.

## Answer — step by step

### Given

Activations = [2,−1,0,3].

Keep probability q = 0.75; keep mask = [1,0,1,1].

### To find

Find training outputs and state the inference rule.

### Formula — which one and why

**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.



$$
a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
$$





$$
a_i^{\mathrm{test}}=a_i
$$



**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.

### Solution — step by step

### Meaning and formula

The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved.



$$
a'_j=\frac{m_j a_j}{q}
$$



The multiplication is entry by entry; there is no dot-product sum.

### Step 1: apply the mask



$$
[1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3]
$$



### Step 2: divide each result by 0.75



$$
a'_1=\frac2{0.75}=2.666666667
$$





$$
a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0
$$





$$
a'_4=\frac3{0.75}=4
$$



**Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero.

### During inference

Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training.

## Drill 42 Mixed dropout and a momentum update

## Question

**Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.

## Answer — step by step

### Given

Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.

Linear-output weights = [0.2,0.3]; bias = 0; target = 1.

Half squared loss; update only the output weights.

Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.

### To find

Find dropped/scaled activations, prediction, gradients, new velocities and new output weights.

### Formula — which one and why

**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.

aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.



$$
a_i^{\mathrm{train}}=\frac{m_ia_i}{q}
$$





$$
a_i^{\mathrm{test}}=a_i
$$



**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

### Formulas used

Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.



$$
a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j
$$



The signed-velocity momentum convention is:



$$
v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}}
$$



### Step 1: apply inverted dropout



$$
a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0
$$



### Step 2: prediction and error



$$
o=0.2(2)+0.3(0)+0=0.4
$$





$$
o-y=0.4-1=-0.6
$$



### Step 3: current weight gradients



$$
g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0
$$



### Step 4: remembered movement plus new correction



$$
v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21
$$





$$
v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09
$$



### Step 5: add velocities to old weights



$$
w_{1,\mathrm{new}}=0.2+0.21=0.41
$$





$$
w_{2,\mathrm{new}}=0.3-0.09=0.21
$$



**Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero.