Learning rate

Also: step size, η, LR

The scale factor on each update θ ← θ − η∇L(θ). Too small and training crawls or stalls; too large and it overshoots, oscillates or diverges.

θ ← θ − η ∇L(θ)

Too small crawls, well-chosen converges, too large overshoots and diverges

Example

On L(θ) = θ² each step multiplies θ by (1 − 2η): η = 0.1 shrinks it by 0.8 per step (1 → 0.8 → 0.64), while η = 1.1 multiplies it by −1.2, so it flips sign and grows without bound.

See it in the lesson: Gradient Descent Variants →

Related terms

More in Optimization & Training