Dropout: Why It Helps Generalization, and the Train/Inference Scaling Trick

Mon, 16 Mar 2026 00:00:00 +0000

Dropout is two ideas bolted together: randomly switch off units during training, then quietly turn them all back on for inference. The interesting parts are (1) why switching units off randomly improves generalization at all, and (2) the fact that “turn them all back on” is not free — the activations come out at the wrong scale unless you correct for it. Getting the scaling wrong is one of the most common deep-learning bugs, so this post works through both, at the same engineering depth as the L1/L2 post.

Dropout on Juntak Noh — AI Notes

Dropout: Why It Helps Generalization, and the Train/Inference Scaling Trick