Inverse Reinforcement Learning
Flipping the RL problem
In standard reinforcement learning you are handed a reward function and asked to find the optimal policy: given R(s, a, s'), solve for π* that maximizes expected discounted return. Inverse RL (IRL) turns this around. You are not given the reward. Instead you are given expert demonstrations — trajectories τ = (s₀, a₀, s₁, a₁, …) produced by someone who already behaves well — and asked to infer the reward function the expert seems to be optimizing.
Standard RL
reward R → find π*
“Here is what is good. Go get it.”
Inverse RL
demonstrations {τ} → infer R (then optionally run RL to get π)
“Watch the expert. Figure out what they want.”
Crucially, IRL recovers a reward, not a policy directly. Once you have the inferred reward you can run any RL solver (value iteration, Q-learning, policy gradient) on it to produce a policy — and that policy should reproduce expert-like behavior.
Why bother? Rewards are hard
Specifying a good reward by hand is surprisingly difficult. A hand-crafted reward for “drive well” or “be a helpful assistant” almost always leaks loopholes the agent will happily exploit (reward hacking). But humans are often great at demonstrating the desired behavior even when they cannot write down the objective. IRL leverages exactly that: it is easier to show than to specify.
IRL vs. behavioral cloning — the key contrast
The naive alternative is behavioral cloning (BC): treat (state → expert action) as a supervised classification problem and imitate the action directly. BC just copies actions. The moment the agent drifts into a state the expert never visited, it has no idea what to do — small errors compound and it falls off-distribution. IRL instead recovers the reason behind the actions (the reward). Because the reward generalizes to unseen states, re-planning under it produces sensible behavior even off the demonstrated path.
In short: BC learns what the expert did; IRL learns why. Learning the “why” transfers far better.
The core difficulty: it is ill-posed
IRL has a fundamental ambiguity: many different reward functions explain the same behavior. The expert's policy is optimal under infinitely many rewards — including the trivial all-zero reward, under which every policy is optimal. So the raw problem is underdetermined (ill-posed); we cannot pin down a unique reward from demonstrations alone.
The fix is to add a prior or regularizer that selects one reward among the many consistent ones. Different IRL methods differ mainly in which principle they use to break the tie.
Maximum-Entropy IRL (Ziebart et al., 2008)
The most influential answer. Among all reward functions (and the policies they induce) that match the expert's feature expectations, prefer the one whose trajectory distribution has maximum entropy — i.e. the least committed, most uniform explanation that still fits the data. This makes the distribution over trajectories P(τ) ∝ exp(Σₜ R(sₜ)): higher-reward trajectories are exponentially more likely, but nothing extra is assumed. Maximizing this likelihood yields a clean gradient.
Feature matching & apprenticeship learning
Most IRL methods assume the reward is linear in features: R(s) = wᵀφ(s), where φ(s) is a feature vector of the state (in our gridworld below, φ is one-hot per cell, so each cell gets its own weight). The expected discounted feature vector of a policy is its feature expectation μ(π) = E[Σₜ γᵗ φ(sₜ)].
Apprenticeship learning (Abbeel & Ng, 2004) makes a clean observation: if your policy's feature expectations match the expert's (μ(π) ≈ μ(expert)), then under any reward linear in those features, your policy performs about as well as the expert. So IRL becomes a matching problem: adjust the reward weights until the induced policy visits states with the same frequencies the expert did. The MaxEnt-IRL update is precisely a gradient toward that match:
where μ_expert comes from the demonstrations and μ_learner comes from the soft-optimal policy under the current reward. When the two feature expectations match, the gradient is zero and the reward has been recovered. This is exactly what the demo below runs.
Interactive demo: gridworld IRL
Draw an expert path from the start (blue dot, bottom-left) to the goal (gold ring, top-right). The demo runs MaxEnt-IRL: it computes the expert's state-visitation features, then iterates the feature-matching gradient to infer a per-cell reward (shown as a heatmap with the reward value in each cell). Finally it runs value iteration on that inferred reward and overlays the recovered greedy policy (arrows). Edit the path and watch both the inferred reward and the recovered policy update.
Draw the expert demonstration: click a cell next to the end of the current path to extend it (click the last cell again to undo a step). The blue dot is the start, the gold ring is the goal. Watch the inferred-reward heatmap and the recovered policy arrows update live.
What you are seeing
Numbers and colors are the inferred reward per cell (warm = high). The dashed line is the expert demonstration. The arrows are the policy recovered by running value iteration on the inferred reward.
The pipeline
demonstration → infer reward → run RL → policy
The recovered arrows trace a path from the start that mirrors the expert's — even though we never told the agent the goal, only showed it where the expert went.
Under the hood: MaxEnt-IRL gradient r ← r + α(f_expert − f_learner) where f are state-visitation features. Path length: 11 cells.
Notice: cells the expert travels through get higher inferred reward, and the recovered policy arrows form a route from start to goal that tracks the demonstration — the agent learned the expert's intent, not just a lookup table of actions. Try drawing a path that detours around the middle: the inferred reward and recovered policy will route around it too.
Connection to RLHF and preference-based reward learning
IRL is the conceptual ancestor of how modern large language models are aligned. RLHF (Reinforcement Learning from Human Feedback) trains a reward model from human preference comparisons — humans rank or pick between model outputs, and a reward model is fit so that preferred responses score higher (typically via a Bradley-Terry preference likelihood). The policy (the LLM) is then optimized against that learned reward with an RL algorithm such as PPO.
This is preference-based reward learning: a cousin of IRL. Both learn a reward signal from human behavior instead of hand-coding it, then run RL on the learned reward. The difference is the form of the human data:
| Aspect | Classic IRL | RLHF / preference learning |
|---|---|---|
| Human input | Demonstrations (expert trajectories) | Comparisons / rankings between outputs |
| What is learned | A reward function | A reward model |
| Then what | Run RL on the reward → policy | Run RL (e.g. PPO) on the reward → aligned LLM |
| Why not hand-code R | Objective hard to specify; easy to demonstrate | “Helpful & harmless” is hard to specify; easy to judge |
The throughline: when an objective is too subtle to write down, learn the reward from human behavior (demonstrations or preferences), then optimize it with RL. That is the shared idea behind IRL, apprenticeship learning, and RLHF.
Common pitfalls
- Reward ambiguity is unavoidable — without a prior (max-entropy, sparsity, linear-in-features) the recovered reward is not unique. Always regularize.
- Feature choice matters enormously — IRL can only express rewards as functions of the features you give it. Bad features → bad reward, no matter the algorithm.
- Inner RL loop is expensive — classic MaxEnt-IRL solves an MDP at every gradient step. Modern variants (e.g. GAIL, adversarial IRL) sidestep this by matching distributions with a discriminator instead.
- Suboptimal experts — if the demonstrations are noisy or imperfect, a model of that noise (such as the Boltzmann/soft policy MaxEnt assumes) is needed, or the inferred reward is biased.
Key Takeaways
- IRL flips standard RL: standard RL maps a reward to a policy; IRL maps expert demonstrations to the reward the expert appears to optimize.
- IRL recovers a reward, not a policy. Once inferred, run any RL solver on it to get a policy that reproduces expert-like behavior.
- It beats behavioral cloning off-distribution. BC copies actions and breaks in unseen states; IRL learns the underlying reward, which generalizes and lets the agent re-plan.
- The problem is ill-posed. Many rewards explain the same behavior, so a prior/regularizer is essential — Maximum-Entropy IRL picks the highest-entropy explanation that still matches the expert.
- Feature matching is the engine. Apprenticeship learning and MaxEnt-IRL adjust reward weights until the policy's feature expectations match the expert's.
- RLHF is preference-based reward learning — the same “learn the reward from human behavior, then run RL” recipe as IRL, using comparisons instead of demonstrations.