# Inverted dropout during training and inference

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s7

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s7/

**Definition.** Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required.

### Explanation and working

Inverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero.

![Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg)

Diagram: Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3.

Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation. Tap the diagram to enlarge.

For activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation.

At inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference.

### Expectation calculation

For a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q:



$$
\mathbb E\left[\frac{ma}{q}\right]=\frac aq\mathbb E[m]=\frac aq\,q=a
$$



The equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction.