“L1 gives you sparse weights, L2 gives you small weights”

What Regularization Actually Is

A model with enough capacity will happily drive its training loss to zero by memorizing the data — including the noise. That is overfitting: great training numbers, bad predictions on anything new. In a linear model it shows up as coefficients that blow up to large, opposing values.

Why does fitting noise require large coefficients? Because of the amplification in the OLS solution \(\hat{\mathbf{w}} = (X^\top X)^{-1}X^\top y\). If \(X^\top X\) has a small eigenvalue (a direction of almost no variance in the data — e.g. two nearly identical features), its inverse has a large eigenvalue, and noise in \(y\) projected onto that direction gets amplified into a large coefficient. Concretely, if \(x_1 \approx x_2\), explaining \(y\) needs only \(w_1=1,\,w_2=0\) — but squeezing out the last bit of noise-fit might use \(w_1=1000,\,w_2=-999\). Since \(x_1 - x_2\) is a near-zero direction, you need enormous coefficients to produce the tiny output that matches the noise. Large \(|\mathbf{w}|\) is the fingerprint of a function contorting itself to fit noise.

Regularization is any technique that deliberately constrains the model so it can’t do that — trading a little training accuracy for better generalization. The flavor we care about here adds a penalty on the size of the weights:

$$ \text{total objective} = \underbrace{L(\mathbf{w})}_{\text{fit the data}} + \underbrace{\lambda\,\Omega(\mathbf{w})}_{\text{keep weights small}} $$

The optimizer now balances two pressures: fit the training data (pushes weights toward whatever explains it, noise included) versus keep weights small (pushes them toward zero). \(\lambda\) sets the exchange rate, and you pick it by cross-validation on held-out data.

The Bias–Variance Tradeoff

Why regularization helps at all comes down to one decomposition. At a point \(x\), the expected test error (squared loss), averaged over random training sets and noise, splits into:

$$ \mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\text{Bias}[\hat{f}(x)]^2}_{\text{systematic error}} + \underbrace{\text{Var}[\hat{f}(x)]}_{\text{wobble}} + \underbrace{\sigma^2}_{\text{irreducible noise}} $$
  • Bias²: how far the average prediction (over datasets) is from the true function. Large when the model is too simple (underfitting).
  • Variance: how much the prediction wobbles as the training set changes. Large when the model is too complex (overfitting).
  • σ²: noise in the data itself (Bayes error) — a floor no model can beat.

The derivation is short. With \(y = f(x) + \varepsilon\), \(\mathbb{E}[\varepsilon]=0\), \(\text{Var}(\varepsilon)=\sigma^2\):

$$ \mathbb{E}[(y-\hat f)^2] = \mathbb{E}[(f - \hat f)^2] + \sigma^2, \qquad \mathbb{E}[(f-\hat f)^2] = (f - \mathbb{E}\hat f)^2 + \mathbb{E}[(\mathbb{E}\hat f - \hat f)^2] = \text{Bias}^2 + \text{Var} $$

(the cross term vanishes because \(\varepsilon\) is independent).

One common confusion worth killing: an estimator being highly data-dependent does not make it biased — that’s variance. They are separate error sources:

  • Unregularized OLS is (under a correctly specified model) an unbiased estimator: its average \(\mathbb{E}[\hat{\mathbf{w}}]\) over datasets lands on the true value → bias ≈ 0. But it swings a lot from dataset to dataset → high variance.
  • Ridge/Lasso pull the coefficients toward zero, shifting the average away from the truth → bias appears. But the wobble shrinks → low variance.

Dartboard analogy: OLS is darts scattered widely but centered on the bullseye (unbiased, high variance); ridge is darts tightly clustered but off-center (biased, low variance).

Since test error is bias² + variance + σ², adding a little bias to cut a lot of variance lowers the total — that’s why regularization works. Against model complexity (or \(1/\lambda\)) the total error is a U-shaped curve, and \(\lambda\) is the knob that slides you along it (\(\lambda\uparrow\) → bias↑, variance↓). The optimal \(\lambda\) sits at the bottom of the U and is found by cross-validation. (Modern caveat: deep nets can break the classic U-curve via double descent, but for the linear/shallow models here the picture holds.)

Setup

We minimize \(\min_{\mathbf{w}} L(\mathbf{w}) + \lambda\,\Omega(\mathbf{w})\), where \(L\) is a data-fit loss (e.g. squared error for linear regression) and \(\Omega\) is one of:

  • L2 (Ridge / weight decay): \(\Omega(\mathbf{w}) = \|\mathbf{w}\|_2^2 = \sum_j w_j^2\)
  • L1 (Lasso): \(\Omega(\mathbf{w}) = \|\mathbf{w}\|_1 = \sum_j |w_j|\)

Where the names come from. L1 and L2 refer to \(L^p\) norms (\(\ell^p\), the L is for Lebesgue). In \(\|\mathbf{w}\|_p = (\sum_j |w_j|^p)^{1/p}\), \(p=1\) is the sum of absolute values (L1) and \(p=2\) is Euclidean length (L2). L2 regularization uses the squared norm \(\sum_j w_j^2\) for differentiability. (Also in the family: \(L_0\) = “count of nonzeros”, \(L_\infty = \max_j|w_j|\).)

The Headline Differences

L2 (Ridge) L1 (Lasso)
Penalty \(\sum_j w_j^2\) \(\sum_j \lvert w_j \rvert\)
Solution Dense — weights shrink but stay nonzero Sparse — many weights become exactly 0
Effect per weight Multiplicative shrinkage Constant-size shrinkage + thresholding
Differentiability Smooth everywhere Non-differentiable at 0
Closed form (linear reg.) Yes: \((X^\top X + \lambda I)^{-1} X^\top y\) No (but efficient solvers exist)
Correlated features Spreads weight across the group Tends to pick one, zeroes the rest
Bayesian prior Gaussian Laplace
Use case Stability, multicollinearity, default choice Feature selection, interpretability, high-dim \(p \gg n\)

Why Each Row Comes Out That Way

Almost every difference in the table branches off one thing: the derivative of the penalty.

Dense (L2) vs. sparse (L1) — the headline. The optimality condition for coordinate \(j\) is “gradient of loss + gradient of penalty = 0.”

  • The L2 penalty \(\lambda w_j^2\) has derivative \(2\lambda w_j\). At \(w_j=0\) this is 0 → no force pushes a coefficient to exactly zero. So even a faint pull from the data moves it off zero to a small nonzero value. Landing at exactly 0 would require the loss gradient to vanish there by coincidence → dense.
  • The L1 penalty \(\lambda|w_j|\) is non-differentiable at 0, so instead of a derivative it has a subgradient interval \([-\lambda, \lambda]\). That creates a threshold: \(w_j=0\) is optimal whenever the loss gradient there is small enough (\(|\partial L/\partial w_j| \le \lambda\)). Only coordinates the data pulls on harder than \(\lambda\) survive; the rest are exactly 0 → sparse.

One-line geometric intuition: viewing the penalty as a “budget” region, L2’s region is a smooth disk/sphere, so the optimal tangency has no reason to land on an axis (a zero coordinate). L1’s region is a diamond with sharp vertices on the axes, so the loss contour tends to hit a vertex (some coordinates zero) first.

Multiplicative shrinkage vs. constant shrinkage + threshold. Under an orthonormal design (\(X^\top X = I\), \(\hat w_j^{\text{OLS}} = (X^\top y)_j\)) it falls out cleanly:

$$ \hat w_j^{\text{ridge}} = \frac{\hat w_j^{\text{OLS}}}{1 + 2\lambda}, \qquad \hat w_j^{\text{lasso}} = \operatorname{sign}(\hat w_j^{\text{OLS}})\,\max\big(|\hat w_j^{\text{OLS}}| - \lambda,\; 0\big) $$

Ridge rescales everything by the same factor (100→33, 0.001→0.0003, nothing hits zero). Lasso shifts everything toward zero by the constant \(\lambda\) and clamps anything that would cross (soft-thresholding) — small coefficients can’t afford the fixed toll and become 0. Both formulas are the direct consequence of the derivative being proportional (ridge) vs. constant (lasso).

Differentiability. \(w^2\) has continuous derivative \(2w\) (smooth). \(|w|\) has left/right derivatives \(-1, +1\) at 0 — a kink. That kink is the source of the threshold and the sparsity.

Closed form or not. L2 + squared loss makes the objective quadratic → setting the gradient to zero is a linear system \((X^\top X + \lambda I)\mathbf{w} = X^\top y\) → closed form. As a bonus, adding \(\lambda I\) makes \(X^\top X\) strictly positive definite, which is why ridge also fixes ill-conditioned/rank-deficient problems. L1 is non-differentiable at 0, so there’s no closed form; you use coordinate descent, LARS, or proximal methods.

Correlated features: spread vs. pick one. Take two near-identical features and a total effect \(c\). L2 cost of splitting is \((c/2)^2 + (c/2)^2 = c^2/2\), cheaper than piling it on one (\(c^2\)) → it splits. L1 cost is \(|c/2| + |c/2| = |c|\), identical to piling on one → it’s indifferent, so the tie breaks arbitrarily and weight concentrates on one (which is why resampling can flip which one survives).

Bayesian prior: Gaussian vs. Laplace. Because the penalty is the negative log-prior — the next section.

The Bayesian View: What a Prior Is

Through Bayes’ theorem, regularization is exactly adding a prior. Bayes says posterior ∝ likelihood × prior:

$$ p(\mathbf{w} \mid \text{data}) \;\propto\; p(\text{data} \mid \mathbf{w})\; \cdot\; p(\mathbf{w}) $$
  • prior \(p(\mathbf{w})\): your belief about the parameters before seeing data (“weights are probably small, near zero”).
  • likelihood \(p(\text{data}\mid\mathbf{w})\): how probable the data is under those parameters.
  • posterior \(p(\mathbf{w}\mid\text{data})\): the updated belief after seeing the data.

MAP (maximum a posteriori) estimation maximizes the posterior = maximizes \((\log\text{likelihood} + \log\text{prior})\). Since \(\log\text{likelihood} \leftrightarrow -L(\mathbf{w})\) (a Gaussian likelihood gives squared error) and \(\log\text{prior} \leftrightarrow -\lambda\Omega(\mathbf{w})\), minimizing \(L(\mathbf{w}) + \lambda\Omega(\mathbf{w})\) is MAP estimation with prior \(\propto e^{-\lambda\Omega(\mathbf{w})}\). The penalty is the negative log-prior.

  • L2 (\(\sum w^2\)) ⟷ Gaussian prior \(\mathcal{N}(0, \sigma^2)\): a smooth bell around 0 — being near zero is cheap, being exactly zero earns nothing special → dense.
  • L1 (\(\sum|w|\)) ⟷ Laplace prior \(\propto e^{-\lambda|w|}\): a sharp peak at 0 with heavier tails. The peak concentrates prior mass at exactly 0 (sparsity), and the heavier tails penalize large weights less than a Gaussian would — matching soft-thresholding’s “kill the small ones, barely touch the large ones.”

The prior’s variance \(\leftrightarrow 1/\lambda\): large \(\lambda\) is a tight prior = “I strongly believe weights are near 0.” So choosing L1 vs. L2 is really choosing what you believe the weights look like (sparse, or small-and-spread).

When and Why Each One Works

Regularization helps when the belief (prior) it encodes is actually true of your problem. The two penalties encode different beliefs, so they succeed in different situations.

Reach for L2 (Ridge) when:

  • Features are correlated or the problem is ill-conditioned. Multicollinearity makes unregularized coefficients huge and unstable; L2’s \(+\lambda I\) term stabilizes the inverse and spreads weight smoothly across correlated features. This is L2’s flagship use case.
  • You believe many features each contribute a little. If the true signal is “dense,” shrinking everything proportionally is right, and zeroing features (L1) would throw away real signal.
  • You just want a stable default. L2 is smooth, has a closed form, and is gradient-descent-friendly. Neural-net “weight decay” is essentially L2.

Reach for L1 (Lasso) when:

  • You believe the true model is sparse — only a handful of features matter. L1 drives the rest to exactly zero, and if that belief holds it recovers the correct support (sometimes with fewer samples than features — compressed sensing).
  • You have more features than data (\(p \gg n\)) and need feature selection. Genomics, huge-vocabulary text, sensor arrays — one fit picks a subset automatically.
  • Interpretability or deployment cost matters. A 50-nonzero model is readable and cheap to run; a dense 10,000-tiny-weight model is neither.

When the bet matches reality you get the bias–variance win essentially for free. When it doesn’t (L1 on a genuinely dense problem), the penalty wrongly zeros real features and you pay in accuracy.

Does It Apply Beyond Linear Models? — MLPs, XGBoost, Random Forests

L1/L2 isn’t specific to linear models; it’s the general idea of penalizing parameter magnitude, so it attaches wherever there are continuous parameters.

  • MLP / neural net: applies directly. Deep learning’s weight decay is L2 (\(\sum w^2\) over all weights). L1 is possible but per-weight sparsity is rarely useful, so it’s less common.
  • XGBoost: uses L1/L2 explicitly — not on coefficients but on the leaf weights (each leaf’s output value). Its objective includes \(\gamma T + \tfrac{1}{2}\,\texttt{reg\_lambda}\sum w^2 + \texttt{reg\_alpha}\sum|w|\) (reg_lambda = L2, reg_alpha = L1, gamma = leaf-count penalty).
  • Random Forest: no magnitude penalty (there are no weights to shrink). It regularizes structurally instead — max_depth, min_samples_leaf, number of trees, bagging, and per-split feature subsampling — achieving the same goal of curbing overfitting.

Two Refinements Worth Knowing

  • Elastic Net — mix L1 and L2: \(\lambda_1\|\mathbf{w}\|_1 + \lambda_2\|\mathbf{w}\|_2^2\). Pros: adds L2’s stability to L1’s sparsity, so a grouping effect keeps correlated features in or out together (stable selection), and it removes pure lasso’s limit of selecting at most \(n\) features when \(p \gg n\). Cons: two hyperparameters to tune, less sparse than pure L1, and the naive version over-shrinks (double shrinkage) and needs a rescaling correction.
  • Relaxed lasso — L1’s constant \(\lambda\) shift biases even the large coefficients it keeps toward zero. Two-stage fix: use L1 only to select the support, then refit the chosen features with no penalty (or a light ridge). You keep L1’s feature selection without its shrinkage bias on the survivors.

Summary

  1. Regularization deliberately constrains model capacity to prevent overfitting; penalizing weight size adds a little bias to cut a lot of variance (the bias–variance tradeoff). It works because in test error = bias² + variance + σ², the variance drop outweighs the bias increase.
  2. L2 rescales coefficients proportionally (dense); L1 shifts by \(\lambda\) and thresholds, sending many to exactly 0 (sparse). The root cause is the penalty’s derivative — 0 at the origin for L2 (no force), a subgradient interval \([-\lambda, \lambda]\) for L1 (a threshold). The same fact shows up geometrically (corners) and as a Bayesian prior (a peaked Laplace).
  3. Choosing: L2 for correlated/dense/stability-as-default; L1 for sparse signal, feature selection, \(p \gg n\); Elastic Net if you want both. The point is that regularization works when the belief it encodes matches reality.