Actor-critic
Also: actor-critic methods, A2C, A3C, advantage actor-critic
The critic learns a value function and supplies an advantage estimate, such as the TD error r + γV(s′) − V(s), which the actor uses in place of the raw return in its policy-gradient update. A2C and A3C run many workers in parallel.
A(s, a) ≈ r + γ V(s′) − V(s)
Example
The critic expects V(s) = 4.5; the agent gets r = 1 and lands in s′ with V(s′) = 5, so with γ = 0.9 the advantage is 1 + 4.5 − 4.5 = +1 and the actor makes that action more likely.