Bellman equation
Also: Bellman equations, Bellman optimality equation, Bellman backup
It expresses V(s) or Q(s, a) in terms of one-step rewards and the values of successor states, weighted by policy and transition probabilities. The optimality version takes a max over actions; repeatedly applying it is the basis of value iteration and Q-learning.
V*(s) = maxₐ Σ_{s′} P(s′ | s, a) [ R(s, a, s′) + γ V*(s′) ]
Example
A deterministic step costs −1 and leads to a state worth 10; with γ = 0.9 the Bellman backup gives V(s) = −1 + 0.9 · 10 = 8.