Dropout: Why It Helps Generalization, and the Train/Inference Scaling Trick
Dropout is two ideas bolted together: randomly switch off units during training, then quietly turn them all back on for inference. The interesting parts are (1) why switching units off randomly improves generalization at all, and (2) the fact that “turn them all back on” is not free — the activations come out at the wrong scale unless you correct for it. Getting the scaling wrong is one of the most common deep-learning bugs, so this post works through both, at the same engineering depth as the L1/L2 post. ...