Evolution Strategies (ES)
Introduction
Evolution Strategies are a family of black-box optimization algorithms for problems where the parameters live in a continuous vector space and we can only evaluate an objective, not differentiate it. Instead of following an analytic gradient, ES maintains a search distribution over parameter vectors — typically a multivariate Gaussian N(θ, Σ) — samples candidate solutions from it, evaluates their fitness, and then nudges the distribution toward the regions that scored well.
Because they only need a fitness number per candidate, ES methods work even when the objective is non-differentiable, noisy, or a complete black box — for example the total reward of a control policy after a full episode in a simulator. The price is sample inefficiency: you learn from scalar feedback rather than from a full gradient, so you generally need many evaluations.
Core loop: sample a population from the search distribution → evaluate fitness → move the distribution toward the best samples (and adapt its shape/spread) → repeat. The distribution "evolves" to concentrate on good solutions.
The Search Distribution
Classic ES uses a Gaussian search distribution centered at a mean vector m (often written θ) with a step size σ:
Each zᵢ is one candidate solution ("offspring"). Here m is the current center of search, σ controls how far we explore, and εᵢ is standard Gaussian noise. We evaluate the objective F(zᵢ) for each offspring, then use those scores to update m and σ for the next generation.
m (mean)
The current best guess / center of the search. Recombination moves it toward high-fitness offspring.
σ (step size)
How far we explore each generation. Large σ → global exploration; small σ → local refinement. Adapted over time.
Σ (covariance)
The full shape of the cloud. CMA-ES adapts Σ to learn correlations and stretch the search along promising directions.
(μ, λ) and (μ + λ) Selection
ES generates a population of λ offspring each generation and keeps the μ best (with μ < λ). The two classic schemes differ in which pool the survivors are chosen from:
| Scheme | Survivors selected from | Behavior |
|---|---|---|
| (μ, λ) | Only the λ new offspring (parents always discarded) | "Comma" — can forget a good parent, but escapes local optima and stale step sizes more easily. Standard in modern ES. |
| (μ + λ) | The μ parents plus the λ offspring | "Plus" — elitist, never loses the best-so-far, but can stagnate because an outdated parent keeps surviving. |
The μ selected offspring are then recombined — usually by (weighted) averaging — to form the mean of next generation's distribution. Notation like (μ/μ_w, λ) denotes weighted recombination over all μ parents, which the demo below uses.
Self-Adaptive Step Size σ
A fixed step size is brittle: too large and the search bounces around the optimum forever; too small and convergence crawls. ES solves this by adapting σ automatically. The intuition behind the classic 1/5-success rule (Rechenberg): if many offspring improve on the parent, the step is too cautious — grow σ; if few do, we are overshooting — shrink σ. A target success rate of about 1/5 keeps progress steady.
In self-adaptive ES, σ (or even a per-coordinate vector of step sizes) is itself encoded in each individual and mutated alongside the parameters. Individuals that happen to carry a well-tuned σ produce better offspring and survive selection, so good step sizes are evolved implicitly — selection tunes the search width for free.
The interactive demo adapts a single global σ toward the observed spread of the selected offspring: as the cloud lands in a tight basin, σ shrinks and the search refines; if the elites are still scattered, σ grows and exploration continues.
Interactive: A Gaussian Search Distribution Climbing a Landscape
Below is a real (μ, λ) Evolution Strategy on a multi-modal 2D fitness landscape (we are maximizing). Each generation it samples λ offspring from N(θ, σ²I), keeps the best half, recombines them into a new mean θ, and adapts σ. Watch the population cloud crawl uphill and the blue search circle shrink as it converges.
Mean θ (search center) Selected (elite) offspring Rejected offspring
The blue circle is the 1σ search distribution N(θ, σ²I). The black ✕ marks the global optimum. Warmer colors = higher fitness.
Controls
Offspring sampled per generation (μ = λ/2 are selected).
Width of the initial Gaussian. Takes effect on Reset / generation 0.
CMA-ES: Learning the Shape of the Landscape
A single scalar σ forces the search cloud to stay a perfect circle. But real landscapes have valleys that run diagonally and ridges that are narrow in one direction and wide in another. CMA-ES (Covariance Matrix Adaptation Evolution Strategy) generalizes the step size into a full covariance matrix Σ, so the Gaussian can become an arbitrarily oriented, stretched ellipse:
where C is the adapted covariance. After each generation CMA-ES updates C using the successful mutation steps — the directions that led to high-fitness offspring — so the ellipse stretches along directions that have been paying off. It also tracks an evolution path (a smoothed history of where the mean has been moving) to detect sustained progress and tune the overall step size. Effectively, CMA-ES learns a local model of the landscape's curvature on the fly, which is why it is one of the most powerful gradient-free optimizers for low-to-medium dimensional continuous problems.
Intuition: a plain ES knows only "how far" to step (σ); CMA-ES also learns in which directions and how those directions correlate — approximating the inverse Hessian without ever computing a derivative.
ES as a Scalable Alternative to RL (OpenAI, 2017)
Salimans et al. showed that a strikingly simple ES can train reinforcement-learning policies competitively with gradient-based methods like policy gradients. The idea is to use a fixed isotropic Gaussian (no covariance adaptation) and treat the population as a way to estimate a gradient by finite differences. With policy parameters θ, perturbation scale σ, and n samples:
The update term (1/(n·σ)) · Σᵢ Fᵢ εᵢ is a finite-difference estimate of the gradient of the smoothed objective 𝔼ε~N(0,I)[ F(θ + σε) ] with respect to θ — it is exactly ∇θ 𝔼[F(θ + σε)]. Notice how it works: directions εᵢ that happen to raise the reward get a large positive weight Fᵢ and pull θ that way; directions that lower reward pull the opposite way. No backpropagation through the policy or the environment is ever needed — only forward evaluations of F. Here α is the learning rate and σ is the (fixed) noise std.
Why it parallelizes trivially: each worker only needs θ and a shared random seed. A worker can regenerate any other worker's perturbation εᵢ from the seed, so workers exchange just scalar rewards (and seeds), not gradients or parameter vectors. That tiny communication footprint let OpenAI scale ES across hundreds or thousands of CPU cores with near-linear speedup — wall-clock training in minutes.
Other advantages: no backprop (works with non-differentiable rewards, long horizons, and sparse/delayed signals), invariance to the action frequency and to monotonic transforms of reward (since it uses fitness ranks in practice), and far better robustness over long episodes. The catch is the same as all ES: sample inefficiency — it needs many more environment interactions than methods that exploit the true policy gradient, trading data efficiency for raw parallel throughput and simplicity.
ES vs Genetic Algorithms vs Gradient Descent
ES sits between two neighbors. Genetic Algorithms (covered in their own lesson) are also population-based and gradient-free, but the mechanics differ; gradient descent is the opposite end — it uses the true analytic gradient.
| Aspect | Evolution Strategies | Genetic Algorithms | Gradient Descent |
|---|---|---|---|
| Representation | Real-valued continuous vectors | Often discrete strings / bit / symbolic genomes | Continuous, differentiable parameters |
| Main operator | Gaussian mutation + recombination; adapt σ / Σ | Crossover-heavy, plus mutation; selection | Step along −∇ of the loss |
| Uses gradient? | No analytic gradient; estimates one by sampling | No; pure selection on fitness | Yes — exact gradient via backprop |
| Adapts search shape | Yes — step size and (CMA-ES) full covariance | Typically not a learned continuous distribution | Via optimizer (momentum, Adam), not a distribution |
| Best when | Continuous, black-box, parallelizable, non-differentiable | Combinatorial / discrete / structured search spaces | Differentiable models with cheap exact gradients |
One-line summary: ES = a Gaussian search distribution over continuous vectors that it shifts and reshapes (σ / covariance adaptation) and that, in the OpenAI formulation, doubles as a sampled gradient estimate. GAs lean on discrete crossover; gradient descent uses the exact derivative.
Tradeoffs & Pitfalls
Strengths
- • No gradients / no backprop required
- • Handles non-differentiable, noisy, black-box objectives
- • Embarrassingly parallel (share seeds + scalar rewards)
- • Adapts its own step size / covariance (CMA-ES)
- • Robust over long horizons and sparse rewards
- • Invariant to monotonic reward transforms (rank-based)
Limitations
- • Sample-inefficient (scalar feedback, not full gradient)
- • CMA-ES scales poorly to very high dimensions (Σ is d×d)
- • Can get trapped on local optima if σ shrinks too early
- • Needs many evaluations — compute-hungry
- • Hyperparameters (λ, σ, weights) still matter
Key Takeaways
- ES is black-box optimization over continuous parameter vectors driven by a Gaussian search distribution N(θ, Σ): sample, evaluate, move toward the best, repeat.
- (μ, λ) selects survivors only from new offspring (non-elitist, escapes traps); (μ + λ) also keeps parents (elitist, never loses the best, can stagnate).
- σ self-adapts — large for exploration, shrinking for refinement; the 1/5 success rule grows or shrinks it based on how many offspring improve.
- CMA-ES adapts the full covariance Σ to learn the local shape of the landscape — directions and correlations — like an estimated inverse Hessian, without any derivatives.
- The OpenAI ES update θ ← θ + α·(1/(nσ))·Σᵢ Fᵢ εᵢ is a finite-difference gradient estimate of 𝔼[F(θ+σε)] — no backprop, trivially parallel via shared seeds, at the cost of sample efficiency.
- Versus GAs (discrete, crossover-heavy) and gradient descent (exact analytic gradient), ES occupies the middle: continuous like SGD, gradient-free like a GA, but uniquely it estimates a gradient and adapts a search distribution.