Dropout is two ideas bolted together: randomly switch off units during training, then quietly turn them all back on for inference. The interesting parts are (1) why switching units off randomly improves generalization at all, and (2) the fact that “turn them all back on” is not free — the activations come out at the wrong scale unless you correct for it. Getting the scaling wrong is one of the most common deep-learning bugs, so this post works through both, at the same engineering depth as the L1/L2 post.

What Dropout Does

During training, a dropout layer takes each unit’s activation \(a\) and, independently, either keeps it (with keep probability \(p\)) or zeroes it (with probability \(1-p\)). Formally, draw a mask \(m \sim \text{Bernoulli}(p)\) per unit and output \(m \cdot a\).

$$ m_i \sim \text{Bernoulli}(p), \qquad \tilde a_i = m_i \, a_i $$

Each forward pass draws a fresh mask, so each minibatch effectively trains a different randomly-thinned version of the network. Backprop only flows through the units that survived; dropped units get zero gradient that step.

Convention warning. I use \(p\) = keep probability throughout (so \(\mathbb{E}[m]=p\)). Many frameworks parameterize by the drop probability instead — PyTorch’s nn.Dropout(p) zeroes each element with probability \(p\), so its keep probability is \(1-p\). Same mechanism, flipped variable. Watch this when reading formulas.

That’s the whole mechanism. Everything below is consequences.

Why It Helps Generalization

There is no single reason; dropout is one of those methods that looks good from several independent angles at once. Here are the ones that actually explain its behavior.

1. It breaks co-adaptation

Without dropout, a hidden unit is free to learn a feature that only works in the presence of some other specific unit — one unit fixes another’s mistakes, and together they encode a fragile, highly specific pattern that often just memorizes training noise. This is co-adaptation.

Dropout makes that strategy impossible. Any given unit might be gone on the next step, so no unit can rely on any particular teammate being present. Each unit is forced to learn a feature that is useful more or less on its own, robust to whichever context happens to survive. The result is more redundant, more independently meaningful features — which is exactly what transfers to unseen data. This was Hinton et al.’s original motivation, and it’s the most intuitive way to see the effect.

2. It is an implicit ensemble of \(2^n\) subnetworks

A layer with \(n\) dropout-able units has \(2^n\) possible masks — that many distinct “thinned” subnetworks. Crucially, they all share the same weights. Each training step samples one subnetwork and nudges the shared weights to make that subnetwork a bit better.

Over training you are jointly fitting an exponentially large ensemble of networks tied together through weight sharing. Ensembling is a classic variance-reduction technique: averaging many models that overfit in different ways cancels out the idiosyncratic errors. Dropout gets that benefit without training or storing \(2^n\) models — the weight sharing collapses the whole ensemble into one set of parameters. As we’ll see, the test-time behavior is precisely an approximation to averaging this ensemble’s predictions, which is what ties this view to the scaling trick below.

3. Multiplicative noise ≈ an adaptive L2 penalty

Multiplying activations by Bernoulli noise is, literally, injecting noise into the hidden representation and asking the model to still get the answer right. Training to be robust to noise is a regularizer.

This can be made precise and it connects straight back to the L1/L2 post. For a linear model (or a GLM) with dropout applied to the inputs, if you average the loss over the dropout noise, the expected objective works out to the ordinary loss plus a penalty of the form

$$ R(\mathbf{w}) \;\propto\; \frac{1-p}{p}\sum_j \big(\text{scale of feature } j\big)\, w_j^2 . $$

That is a ridge (L2) penalty — but one whose per-weight strength is automatically proportional to each feature’s own scale/variance. In other words, dropout behaves like ridge applied to normalized features: it self-adjusts the penalty so rarely-active or small-scale features aren’t under-penalized and dominant features aren’t over-penalized. The keep rate \(p\) plays the role of \(1/\lambda\): smaller \(p\) (more aggressive dropping) ⟹ larger \(\frac{1-p}{p}\) ⟹ stronger regularization. (Wager, Wang & Liang, 2013, worked out this equivalence.)

The bias–variance summary

All three views are the same trade in the language of the bias–variance decomposition: each sampled subnetwork is individually weaker (a bit more bias), but forcing robust, redundant features and averaging an ensemble cuts variance substantially. Net test error goes down. Dropout is a knob on the same curve L1/L2 sit on — just implemented by stochastic masking instead of an explicit penalty term.

Train vs. Inference: The Scaling Problem

Here is the part everyone eventually trips on.

At training time, a unit downstream of dropout receives, on average, only a fraction \(p\) of the signal it would receive with all units on — because each incoming activation is present only with probability \(p\):

$$ \mathbb{E}[\tilde a_i] = \mathbb{E}[m_i]\, a_i = p \, a_i . $$

So every layer learns its weights against inputs whose expected magnitude is scaled by \(p\).

At inference time, we don’t want random predictions, so we stop masking and use the full network (all units on). But now each unit passes its full activation \(a_i\) downstream — roughly \(1/p\) times larger than the \(p\,a_i\) the next layer was trained to expect. Every subsequent layer sees inputs at the wrong scale, the distribution shifts, and the outputs are garbage. We have to correct for the factor of \(p\).

There are two places to put the correction.

Classic dropout: scale down at test time

Train exactly as above (output \(m_i a_i\), no scaling). At test time, multiply activations by \(p\) so the deterministic activation matches the training-time expectation:

$$ a_i^{\text{test}} = p \, a_i = \mathbb{E}_{\text{train}}[m_i a_i]. \quad\checkmark $$

Equivalently, multiply the outgoing weights by \(p\). This works, but it makes the inference code depend on \(p\) and differ from a plain network.

Inverted dropout: scale up at train time (the modern default)

Do the correction during training instead, by dividing the kept activations by \(p\):

$$ \tilde a_i = \frac{m_i \, a_i}{p}, \qquad \mathbb{E}[\tilde a_i] = \frac{p \, a_i}{p} = a_i . $$

Now the training-time expectation already equals the plain, unscaled activation — so at inference you do nothing at all: just use \(a_i\) with all units on. This is what PyTorch, TensorFlow, etc. actually implement, and it’s the default for good reasons:

  • Inference code is identical to a network without dropout. In eval mode the dropout layer is a literal no-op (identity). Deployment doesn’t need to know \(p\).
  • The dropout rate is decoupled from inference. You can change \(p\), or turn dropout off entirely, without touching a single line of inference code.
  • All the extra cost lives at training time, where you’re already doing the expensive work; the inference path stays maximally simple and fast.
Classic dropout Inverted dropout (default)
Train output \(m_i \, a_i\) \(m_i \, a_i \,/\, p\)
Test output \(p \, a_i\) (scale down) \(a_i\) (no change)
Where the \(1/p\) lives test time train time
Inference code depends on \(p\) same as no-dropout

Numeric example (\(p = 0.5\), i.e. drop half the units): under inverted dropout, every surviving activation is multiplied by \(1/p = 2\) during training, so the expected total signal into the next layer is preserved. At test time you feed activations straight through, unscaled. Under classic dropout you’d instead leave training alone and halve every activation at test.

The model.eval() bug. Because train and inference behave differently, the framework needs to know which mode it’s in. In PyTorch that’s model.eval() (and model.train() to switch back); it flips Dropout to its identity path (and BatchNorm to running stats). Forgetting model.eval() at inference is a classic bug: dropout stays active, randomly zeroing units, so your predictions become noisy and worse — and non-deterministic across runs. If your eval accuracy is mysteriously low and jittery, check this first.

Why scaling is exactly the ensemble average (for the linear part)

The scaling isn’t an arbitrary patch — it’s what makes the single deterministic forward pass approximate the ensemble from view #2. The true ensemble prediction averages over masks, \(\mathbb{E}_m[f(\tilde{\mathbf a})]\). For any linear operation the expectation passes straight through: \(\mathbb{E}_m[\tilde{\mathbf a}]\cdot W = \mathbf a \cdot W\) exactly (that’s precisely what the \(1/p\) scaling arranges). So the deterministic full-network pass computes the network evaluated at the expected activation, which equals the ensemble mean for the linear parts and closely approximates it through the nonlinearities (Jensen’s inequality makes it inexact, but empirically tight). Srivastava et al. show this deterministic pass computes, in effect, the normalized geometric mean of the \(2^n\) subnetworks’ output distributions — exactly for a single linear/softmax layer, approximately for deep nets. The scaling and the ensemble view are the same fact.

Practical Notes

  • Typical rates. The original paper’s default is drop-half (\(p=0.5\) keep) on fully-connected hidden layers, with a lighter touch (keep \(0.8\)–\(0.9\)) on inputs. More dropping = stronger regularization; tune it like any regularization knob.
  • Convolutional layers. Plain dropout is weaker on conv feature maps because neighboring activations are spatially correlated — zeroing one pixel barely removes information its neighbors still carry. SpatialDropout (drop whole channels) and DropBlock (drop contiguous regions) fix this by removing correlated groups at once.
  • Interaction with BatchNorm. BatchNorm is itself a strong regularizer, and stacking dropout before it can cause a “variance shift” between train and test that hurts (Li et al., 2019). Many modern CNNs (e.g. ResNets) lean on BatchNorm with little or no dropout. When you use both, order and placement matter.
  • Transformers use dropout heavily — on attention weights, on residual branches, and on embeddings — because those layers are wide, mostly fully-connected, and prone to overfitting; the conv-style spatial-correlation caveat doesn’t apply.
  • MC Dropout (a feature, not a bug). If you deliberately keep dropout on at inference and run many forward passes, the spread of predictions estimates model uncertainty (Gal & Ghahramani, 2016). It’s the one time leaving dropout on at test is correct — and it’s literally the model.eval() bug repurposed on purpose.

Summary

  1. Why it generalizes. Dropout stops units from co-adapting into fragile joint features, trains an implicit weight-shared ensemble of \(2^n\) subnetworks, and injects multiplicative noise that acts like an adaptive L2 penalty. In bias–variance terms: slightly more bias per subnetwork, much less variance — net win.
  2. Train vs. inference. Training masks units, so downstream activations are scaled by \(p\) in expectation. Inference uses the full network, so that factor of \(p\) must be corrected or every layer sees the wrong input scale.
  3. The scaling trick. Inverted dropout (the default) divides kept activations by \(p\) during training, making inference a plain, unscaled, all-units-on forward pass — identical to a network with no dropout. Remember model.eval() so the framework actually switches to that path.