Deep Learning · TEE · section

Inverted dropout during training and inference

Regularization and generalization

Definition. Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required.

Explanation and working

Inverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero.

Diagram: Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3.

For activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation.

At inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference.

Expectation calculation

For a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q:

LaTeX: \mathbb E\left[\frac{ma}{q}\right]=\frac aq\mathbb E[m]=\frac aq\,q=a

The equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction.