L1/L2 Regularization
Regularization prevents overfitting by adding a penalty term to the loss function that discourages large weights. L1 regularization promotes sparsity, while L2 regularization promotes small but non-zero weights.
Mathematical Formulation
L1 Regularization (Lasso)
Loss = Original Loss + λ × Σ|wᵢ|
Gradient: ∂Loss/∂w = ∂(Original Loss)/∂w + λ × sign(w)
L2 Regularization (Ridge)
Loss = Original Loss + λ × Σwᵢ²
Gradient: ∂Loss/∂w = ∂(Original Loss)/∂w + 2λw
λ (Lambda)
Regularization strength - higher values mean stronger regularization
Interactive Visualization
Why L1 Creates Sparsity (Geometry)
Regularization is equivalent to constraining the weights to a region around the origin. L1 (Σ|wᵢ| ≤ t) is a diamond whose corners lie exactly on the axes; L2 (Σwᵢ² ≤ t) is a circle. The solution is where the loss contours (the orange ellipses, centred at the unregularized optimum ŵ) first touch the region. Because the diamond has sharp corners on the axes, the touch point very often lands on a corner — driving one weight to exactly zero. The smooth circle has no corners, so L2 shrinks weights but rarely sets any to zero.
Shrink the budget and watch the L1 solution snap onto an axis (a zero weight) while the L2 solution slides smoothly along the circle, keeping both weights non-zero.
When to Use Each Type
| Aspect | L1 Regularization | L2 Regularization |
|---|---|---|
| Feature Selection | Yes - drives weights to exactly zero | No - shrinks weights but keeps them non-zero |
| Sparsity | Creates sparse models | Keeps all features |
| Computational Efficiency | More efficient for sparse data | Generally faster gradient computation |
| Interpretability | Better - fewer active features | Harder - all features contribute |
| Robustness | Less robust to outliers | More robust to outliers |
Pros and Cons
L1 Regularization
Pros:
- Automatic feature selection
- Creates interpretable models
- Good for high-dimensional data
- Reduces model complexity
Cons:
- Can be unstable with correlated features
- Non-differentiable at zero
- May discard useful features
- Less smooth optimization
L2 Regularization
Pros:
- Smooth and differentiable everywhere
- Stable with correlated features
- Better for prediction accuracy
- Computationally efficient
Cons:
- Doesn't perform feature selection
- All features remain in the model
- Less interpretable
- May not work well with irrelevant features
Practical Tips
- • Start with L2 regularization as it's generally more stable
- • Use L1 when you suspect many features are irrelevant
- • Try Elastic Net (combines L1 and L2) for the best of both worlds
- • Always use cross-validation to tune the regularization parameter λ
- • Standardize features before applying regularization
- • Monitor both training and validation loss to detect overfitting