Gradient Descent Variants
Introduction
Almost every model you train — from linear regression to a giant neural network — is fit by gradient descent. The idea is simple: you have a loss function L(θ) that measures how wrong the model is for its parameters θ, and you repeatedly nudge θ in the direction that makes the loss smaller. Do that enough times and you roll downhill into a minimum.
The catch is how you take those steps. The plain version is slow and fragile on the ravine-shaped loss surfaces that real problems produce. Decades of work gave us better steppers — momentum, RMSProp, and Adam — that converge faster and are far less sensitive to how you tune them. This lesson builds each one up from the same core update rule, and lets you race them head-to-head on a tricky 2D surface.
The update rule: move downhill
The gradient ∇L(θ) is the vector of partial derivatives of the loss with respect to every parameter. It points in the direction of steepest increase of the loss. We want to decrease the loss, so we step in the opposite direction — we subtract the gradient:
- θ — the parameters (weights) being optimized.
- ∇L(θ) — the gradient of the loss; points uphill.
- η (eta) — the learning rate, the step size.
- The minus sign is the whole point: subtracting the uphill direction moves us downhill.
Common mistake: writing θ ← θ + η∇L. That is gradient ascent — it climbs the loss and your model gets worse every step. Descent always subtracts the gradient.
A small worked check: if L(θ) = θ², then ∇L = 2θ. Starting at θ = 1 with η = 0.1, the update is θ ← 1 − 0.1·(2·1) = 0.8, then 0.64, 0.512, … sliding toward the minimum at θ = 0. The minus sign is what pulls it inward.
The learning rate η: the most important knob
η controls how big each step is. It is usually the single most important hyperparameter to get right, and there is a real Goldilocks zone:
η too small
Tiny steps. Training converges but agonizingly slowly — you may need millions of iterations, and it can stall on flat regions before reaching the minimum.
η just right
Steps are large enough to make fast progress but small enough to keep descending smoothly toward the minimum.
η too large
Steps overshoot the minimum. The loss bounces back and forth across the valley (oscillates), and if η is large enough each step lands further out than the last — the loss diverges to infinity.
You will see all three behaviors in the demo below: drag the η slider up with plain SGD and watch the path go from a slow crawl, to a clean descent, to a wild zig-zag that flies off the surface.
Interactive: the optimizer race
Below is a real loss surface, L(x, y) = (1 − x)² + 6(y − x²)², a curved, narrow valley (a scaled Rosenbrock function) with its global minimum at (1, 1). This shape is notoriously hard: the floor of the valley is nearly flat along its length but steep across its width, so a single step size cannot suit both directions at once. Pick an optimizer and learning rate, click to set a start point, and the demo runs that optimizer's actual update equations — with analytic gradients — and traces the iterates over the contours. Add several runs to overlay and compare them.
Click anywhere on the surface to move the start point. Lighter regions are low loss (the curved valley); the bullseye is the global minimum at (1, 1). The dashed line previews the selected optimizer; press Run / Add to race it.
Optimizer
Things to try
• Race SGD vs Adam at η = 0.02 — watch SGD crawl along the valley.
• Push η toward 0.2 with SGD: it overshoots and diverges.
• Momentum: see it overshoot, then settle.
• RMSProp / Adam stay stable across a much wider η range.
Batch, stochastic, and mini-batch
Where does the gradient come from? The loss is an average over your training examples, so the true gradient is also an average over all of them. How many examples you use to estimate it each step defines three flavors of gradient descent:
| Variant | Examples per step | Gradient | Tradeoff |
|---|---|---|---|
| Batch (full) | All N examples | Exact, smooth | Stable path, but each step is expensive; one update per full pass. |
| Stochastic (SGD) | 1 example | Very noisy estimate | Cheap, many fast updates; noisy path that can escape shallow traps but jitters near the minimum. |
| Mini-batch | A small batch (e.g. 32–256) | Moderately noisy | The sweet spot: stable enough, cheap enough, and uses hardware (GPU) efficiently. |
The noise/speed tradeoff: bigger batches give a more accurate gradient and a smoother path but cost more per step; smaller batches are cheap and their noise can even help jump out of poor regions, but the path wanders. Mini-batch gradient descent is what essentially all deep learning uses in practice. (Confusingly, "SGD" in frameworks usually means mini-batch SGD.)
Momentum: build up velocity
On a ravine, plain SGD wastes most of its motion bouncing across the steep walls and barely advances along the gentle floor. Momentum fixes this by accumulating a running average of past gradients — like a heavy ball rolling downhill that keeps its inertia:
- v — the velocity, an exponentially decaying running sum of gradients.
- β — the momentum coefficient (commonly 0.9), how much past velocity is retained.
Why it helps: along the valley floor the gradient consistently points the same way, so those contributions accumulate and velocity grows — the ball accelerates. Across the valley the gradient flips sign every step, so those contributions cancel in the running average, damping the oscillation. The net effect is faster progress in the useful direction and a smoother path. The price is that the accumulated velocity can carry it past the minimum, so it may overshoot and swing back before settling — visible in the demo.
Adaptive learning rates: RMSProp and Adam
Momentum still uses one global η for every parameter. But different parameters can need very different step sizes. Adaptive methods give each parameter its own effective learning rate based on the history of its gradients.
RMSProp
RMSProp keeps a running average of the squared gradient for each parameter, then divides the step by its square root. Directions with large, persistent gradients get scaled down; small, flat directions get scaled up — so every coordinate moves at a sensible pace.
- s — running average of squared gradients (per parameter).
- ρ — decay rate for that average (commonly 0.9); (∇L)² is element-wise.
- ε — a tiny constant (e.g. 10⁻⁸) to avoid dividing by zero.
Adam = Momentum + RMSProp
Adam (Adaptive Moment Estimation) combines both ideas: it keeps a momentum-style running mean of the gradient (first moment m) and an RMSProp-style running mean of the squared gradient (second moment s). Because both start at zero they are biased toward zero early on, so Adam applies a bias correction before using them:
- β₁ ≈ 0.9 — decay for the first moment (the momentum part).
- β₂ ≈ 0.999 — decay for the second moment (the RMSProp part).
- t — the step number; β₁ᵗ and β₂ᵗ shrink toward 0, so the bias-correction factors approach 1 as training proceeds.
- Typical defaults: η = 0.001, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸.
Adam is the default optimizer for most deep learning today because it converges quickly and works well across a wide range of learning rates with little tuning. RMSProp and momentum-SGD remain strong choices — well-tuned SGD with momentum sometimes generalizes slightly better on vision tasks.
Minima, saddle points, and convexity
Gradient descent only ever follows the local slope, so where it ends up depends on the shape of the loss surface:
Convex losses
A convex loss is bowl-shaped with a single minimum. Here gradient descent (with a sensible learning rate) is guaranteed to find the global minimum. Linear and logistic regression with standard losses are convex.
Non-convex losses
Neural-network losses are highly non-convex, riddled with local minima and saddle points (flat spots that go up in one direction and down in another). Descent can stall there. In high dimensions saddle points, not bad local minima, are the main obstacle — and momentum / adaptive methods help power through them.
Reassuringly, in large neural networks most local minima turn out to give similarly good loss, so finding the exact global minimum is rarely necessary — a good-enough minimum found quickly is what these optimizers deliver.
At a glance
| Optimizer | Key idea | State kept | Strength |
|---|---|---|---|
| SGD | θ ← θ − η∇L | None | Simple; strong baseline when tuned |
| Momentum | Accumulate a velocity v | Velocity (1×) | Accelerates, damps oscillation |
| RMSProp | Per-parameter lr from √(avg g²) | Squared-grad avg (1×) | Adapts step size per direction |
| Adam | Momentum + RMSProp + bias correction | m and s (2×) | Fast, robust default |
Key Takeaways
- The core rule is θ ← θ − η∇L(θ): you subtract the gradient to move downhill. Adding it would be ascent.
- The learning rate η is the key knob: too small is painfully slow, too large overshoots, oscillates, and can diverge.
- Batch uses all data per step (smooth, costly); SGD uses one example (noisy, cheap); mini-batch is the practical sweet spot for the noise/speed tradeoff.
- Momentum accumulates a velocity v ← βv + ∇L, then θ ← θ − ηv — it accelerates along consistent directions and damps oscillation across the valley.
- RMSProp gives each parameter an adaptive learning rate by dividing the step by the root of a running average of squared gradients.
- Adam combines momentum and RMSProp with bias correction; it is the robust default for deep learning.
- For convex losses gradient descent finds the global minimum; for non-convex neural nets it finds a good local minimum, navigating saddle points along the way.