Everyone can recite “high bias is underfitting, high variance is overfitting.” Far fewer can look at a training run and say which one they have and what to do about it. This post does both: first the bias–variance decomposition tightly enough to be useful, then a practical playbook — learning curves, the train/validation gap, cross-validation, and the traps — for diagnosing overfitting on a real model. It’s the diagnostic companion to the L1/L2 and dropout posts, which cover the fixes.
The Decomposition
Take a target \(y = f(x) + \varepsilon\) with noise \(\mathbb{E}[\varepsilon]=0\), \(\text{Var}(\varepsilon)=\sigma^2\). Train a model \(\hat f\) on a random dataset and ask for its expected squared error at a point \(x\), averaged over both the noise and the randomness of which training set you happened to draw:
$$ \mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\big(f(x) - \mathbb{E}[\hat f(x)]\big)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\big[(\hat f(x) - \mathbb{E}[\hat f(x)])^2\big]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{irreducible}} $$- Bias² — how far the average model (over many training sets) sits from the truth. This is the error from your model being too rigid to represent \(f\). Underfitting.
- Variance — how much the model wobbles from one training set to the next. This is the error from your model being flexible enough to chase the particular noise in this dataset. Overfitting.
- σ² — noise in the labels themselves. No model beats this floor; it’s the Bayes error.
The tradeoff: as you increase model complexity, bias falls (the model can represent more) but variance rises (it has more freedom to fit noise). Total error is a U-shaped curve in complexity — too simple and bias dominates, too complex and variance dominates, and the sweet spot is in between. Every knob you own (model size, features, regularization strength, training time) moves you along this curve.
The key mental model. Bias is a systematic miss — wrong on average, the same way every time. Variance is instability — right on average but flailing around. A model can be highly data-dependent (high variance) yet unbiased. These are two different diseases with two different cures, which is exactly why diagnosing which one you have matters.
What Each One Looks Like
| High bias (underfitting) | High variance (overfitting) | |
|---|---|---|
| Training error | High | Low (often near zero) |
| Validation error | High, ≈ training error | High, ≫ training error |
| Train–val gap | Small | Large |
| More training data | Doesn’t help | Helps |
| Symptom | Model too simple / over-regularized | Model too complex / under-regularized |
The single most informative number is the gap between training and validation error. Its size tells you variance; the level of the training error tells you bias. Read them together.
Practical Diagnosis
1. The train/validation gap (start here)
Split off a validation set the model never trains on, and compare:
- Training error high, validation error about the same → high bias. The model can’t even fit the data it has seen. More data or more regularization won’t help; you need a more capable model.
- Training error low, validation error much higher → high variance. The model nailed the training set and failed to generalize. This is textbook overfitting.
- Both low and close → you’re in good shape; consider whether you can push complexity for more.
To make “high” and “low” concrete, compare training error against an irreducible-error proxy — human-level performance, a strong baseline, or a known Bayes error. This is Andrew Ng’s framing:
$$ \underbrace{\text{avoidable bias}}_{\text{train err} - \text{Bayes err}} \quad\text{vs.}\quad \underbrace{\text{variance}}_{\text{val err} - \text{train err}} $$Whichever gap is larger is the problem to attack first. A model at 8% train / 10% val error is a bias problem if humans hit 1%, but a variance problem if humans hit 7.5%. The same raw numbers, opposite diagnoses — which is why an absolute target (Bayes proxy) matters.
2. Learning curves (error vs. training-set size)
Train the model on growing subsets of the data (say 10%, 20%, …, 100%) and plot training error and validation error against the number of samples. The shapes are diagnostic:
- High bias: training error rises to a high plateau as you add data (a bigger set is harder to fit perfectly), validation error falls to meet it, and the two converge close together — but at a high error. The curves have already flattened, so the tell is: adding more data won’t help. You need a bigger model or better features.
- High variance: training error stays low, validation error is much higher, and the gap is wide but still closing as data grows. The tell: more data is still helping — the validation curve is trending down. Collect more data, augment, or regularize.
Learning curves are the single best tool because they separate “would more data help?” from “would a better model help?” — a question the train/val gap alone can’t fully answer.
3. Validation (complexity) curves
Fix the data and sweep one hyperparameter that controls capacity — polynomial degree, tree depth, number of layers/units, or regularization strength \(\lambda\). Plot train and validation error against it:
- Training error decreases monotonically as complexity rises.
- Validation error is U-shaped: it falls, bottoms out, then rises as overfitting sets in.
- The bottom of the validation U is your operating point; everything to the right of it is overfitting, everything to the left is underfitting.
For regularization specifically, sweeping \(\lambda\) traces this U directly — small \(\lambda\) = high variance (right side), large \(\lambda\) = high bias (left side).
4. Cross-validation (make the estimate trustworthy)
A single validation split is itself a noisy estimate — with a small or unlucky split you might diagnose the split, not the model. k-fold cross-validation rotates the validation fold through all \(k\) parts and averages, giving a lower-variance estimate of generalization error and, as a bonus, a standard deviation across folds. That spread is itself a diagnostic: wildly different scores across folds is a sign of high variance (or too little data, or a leaky/heterogeneous split).
Match the CV scheme to the data:
- Stratified k-fold for classification (preserve class balance in each fold).
- Group k-fold when rows cluster (same patient, user, or document across rows) — never let the same group land in both train and validation, or you leak.
- Time-series split for temporal data — always train on the past and validate on the future; a random shuffle here leaks the future into training and gives a fantasy score.
5. Loss curves over training (for iterative models)
For anything trained by gradient descent, plot train and validation loss against epochs. The overfitting signature is unmistakable: training loss keeps decreasing while validation loss bottoms out and starts climbing back up. The bottom of the validation curve is your early-stopping point — and “watch val loss, stop when it turns up” is itself one of the cheapest regularizers you have.
Remedies — Once You Know Which One
Diagnosis is only useful because the two problems have opposite fixes. Applying the wrong one makes things worse (e.g. adding regularization to an underfit model).
| Reduce bias (fix underfitting) | Reduce variance (fix overfitting) |
|---|---|
| Bigger / more expressive model | More training data |
| Add features, richer representations | Data augmentation |
| Train longer, lower the learning-rate floor | Regularization (L1/L2), dropout |
| Decrease regularization \(\lambda\) | Increase regularization \(\lambda\) |
| Better architecture / less aggressive pooling | Simpler model, fewer features |
| — | Early stopping |
| — | Ensembling / bagging |
Note the symmetry on the \(\lambda\) row: the same knob is turned in opposite directions depending on the diagnosis. That’s the whole reason you diagnose first.
Traps That Fake a Good (or Bad) Diagnosis
The methods above only work if the validation number actually reflects generalization. These are the ways it silently doesn’t:
- Data leakage. Fitting preprocessing (scaling, imputation, feature selection, target encoding) on the full dataset before splitting leaks validation information into training → optimistically low validation error, then a nasty surprise in production. Fit every transform inside the CV loop, on training folds only.
- Target leakage. A feature that encodes the label (or something only available after the outcome) gives near-perfect validation scores and useless production performance. If a result looks too good, hunt for a leaked feature first.
- Tuning on the test set. Every hyperparameter you choose by looking at a set makes that set optimistic. Keep a final test set you touch exactly once; do all tuning on validation/CV.
- Validation set too small. A 200-example val set gives a noisy accuracy (±several %); you may be reacting to sampling noise. Use CV or a larger holdout.
- Distribution shift / non-iid data. If validation is drawn iid from training but production isn’t (new time period, new users, new geography), a great val score still overfits to the past distribution. Validate on data that resembles deployment.
- Double descent (the modern caveat). For heavily overparameterized models, test error can fall, rise (the classic U), then fall again past the interpolation threshold. The “bigger model always overfits” intuition breaks in this regime — very large neural nets often generalize well despite fitting training data perfectly. The diagnostic tools still work; the monotone “complexity ↑ ⟹ variance ↑” story is what needs an asterisk.
A Minimal Workflow
- Hold out a test set. Don’t touch it until the very end.
- Get a training/validation split (or k-fold CV) with the right scheme (stratified / group / time-series).
- Compare train error, val error, and a Bayes-error proxy → decide bias vs. variance.
- If ambiguous, plot a learning curve → does more data help?
- Apply the matching remedy from the table; for iterative models, watch val-loss curves and early-stop.
- Re-diagnose. Iterate. Evaluate on the test set once, at the end, to report honest generalization.
Summary
- Bias is systematic error (underfitting, high train and val error); variance is instability (overfitting, low train error but a large train–val gap). Total error is bias² + variance + irreducible noise, and complexity trades one for the other along a U-curve.
- Diagnose with the train/val gap plus an absolute baseline (Bayes/human error): avoidable bias = train − Bayes, variance = val − train; attack the larger one. Confirm with learning curves (does more data help?), validation curves (where’s the U’s bottom?), and cross-validation (is the estimate stable?).
- The two problems have opposite cures, so diagnosis must come before treatment — and the whole thing is only valid if you’ve avoided leakage, kept an untouched test set, and validated on data that looks like deployment.