LSTM
Also: long short-term memory
Long short-term memory adds a cell state that is updated additively and controlled by three sigmoid gates. Because the forget gate can hold the cell nearly unchanged, gradients survive across many more time steps than in a plain RNN.
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ, hₜ = oₜ ⊙ tanh(cₜ)
Example
With 100-dim inputs and a 128-dim state, an LSTM layer has 4 · (128·228 + 128) = 117,248 parameters, four sets of weights for its three gates and candidate update.