Adam
Also: Adam optimizer, AdamW, Adaptive Moment Estimation
Adam keeps running averages of the gradient (m) and its square (s), corrects both for their zero start, and steps by η·m̂ / (√ŝ + ε). Typical defaults: η = 0.001, β₁ = 0.9, β₂ = 0.999.
m ← β₁m + (1 − β₁)g; s ← β₂s + (1 − β₂)g²; θ ← θ − η m̂ / (√ŝ + ε)
Example
On the first step bias correction gives m̂ = g and ŝ = g², so the update is η·g/|g|: every parameter moves by about η = 0.001, whether its gradient is 0.0001 or 1,000.