Learning rate schedule
Also: learning rate scheduling, learning rate decay, learning rate warmup, cosine annealing
A large rate early covers ground and a small rate late settles precisely into a minimum, which a fixed rate cannot do. Common choices: step decay, cosine annealing, linear warmup then cosine, and one-cycle.
η(t) = η_min + ½(η_max − η_min)(1 + cos(π·t / T))
Example
Cosine annealing with η_max = 0.1, η_min = 0 and T = 100 steps gives η = 0.1 at step 0, 0.05 at step 50 and 0 at step 100.