Momentum
Also: SGD with momentum, heavy-ball momentum, Nesterov momentum
Instead of stepping along the current gradient, update a velocity v ← βv + ∇L and step θ ← θ − ηv. Consistent gradients accumulate, oscillating ones cancel; β is commonly 0.9.
v ← β v + ∇L(θ); θ ← θ − η v
Example
With β = 0.9 and a gradient that stays constant at g, the velocity builds up to g / (1 − 0.9) = 10g, so steps along a steady valley floor become up to 10× larger than plain SGD's.