Autonomous Learning

When the agent decides what to learn

Most machine learning is passive: a human assembles a dataset, labels it, and the model learns from whatever it is handed. Autonomous learning flips this around. Here the agent decides what to learn and how to gather its own experience, with little or no supervision. It chooses where to look, which situations to put itself in, and which problems to tackle next.

This matters most in open-ended environments where there is no fixed dataset and rewards are sparse or absent — a robot in a new room, an agent dropped into a game world, a research assistant exploring an unfamiliar codebase. The central question becomes: given limited supervision, how should an agent spend its own time?

Passive vs. autonomous learning

The distinction is about who controls the data stream, not which algorithm runs underneath.

Intrinsic motivation and curiosity

In reinforcement learning, an agent maximises reward. But what if the environment gives almost no reward — Montezuma's Revenge, where you must cross several rooms before any points appear? A purely reward-seeking agent flails, because random actions almost never stumble onto the rare payoff. Intrinsic motivation manufactures an internal reward signal that pushes the agent to explore even when the environment is silent.

The agent optimises a combined reward r = r_extrinsic + β · r_intrinsic, where the intrinsic term rewards novelty or surprise. Two classic recipes:

• Count-based bonuses. Give a bonus that shrinks as a state is visited more often, e.g. r_intrinsic(s) = 1/√(N(s)) where N(s) is the visit count of state s. Rarely-seen states are worth more, so the agent is pulled toward unexplored territory. In huge state spaces, N(s) is approximated with a learned density model ("pseudo-counts").
• Prediction-error / curiosity (ICM). The Intrinsic Curiosity Module trains a forward model that predicts the next state; the intrinsic reward is the model's prediction error. States the agent cannot yet predict are surprising — and therefore rewarding to visit. To avoid being surprised by irrelevant pixel noise, ICM predicts in a learned feature space that only keeps what the agent's actions can affect.

Interactive demo: curiosity vs. random exploration

The agent (red dot) explores a two-room gridworld joined by a single doorway. It keeps a visit-count map (the heatmap), and you choose what drives its movement:

• Random walk — each step picks a random open neighbour. This is the uninformed baseline.
• Curiosity — each step moves to the neighbour with the highest novelty bonus 1/√(count+1), i.e. the least-visited adjacent cell (ties broken with a seeded coin flip).

Automatic curriculum generation

Curiosity decides where to look within a task. A curriculum decides which tasks to attempt, and in what order. An autonomous learner can propose its own sequence of progressively harder goals — staying in the sweet spot where a challenge is neither trivially easy nor impossibly hard (the "zone of proximal development").

• Goal generation. A goal-conditioned agent samples goals it can reach sometimes but not reliably — goals of intermediate difficulty give the strongest learning signal. As the agent improves, the proposed goals automatically get harder.
• POET (Paired Open-Ended Trailblazer). Co-evolves a population of environments alongside the agents that solve them. New environments are mutated from old ones, kept only if they are not too easy and not too hard for the current population, and solutions are transferred between environments. The curriculum is generated and never "finished" — it keeps inventing new challenges.

Open-ended learning and skill discovery

Rather than mastering one task, an open-ended learner accumulates a repertoire of reusable skills that can be recombined to solve new problems. This shifts the goal from a single policy to an ever-growing library of abilities.

A vivid recent example is Voyager, an LLM-driven agent in Minecraft. It has no fixed objective; instead it:

• uses an LLM to propose its own next task based on what it has and where it is (an automatic curriculum);
• writes each learned behaviour as a small program and stores it in a growing skill library, keyed by a description so it can be retrieved later;
• composes existing skills to bootstrap harder ones — "mine wood" → "craft a table" → "craft a pickaxe" → "mine stone" — so capability compounds over time rather than restarting from scratch.

More generally, unsupervised skill discovery methods learn a set of distinguishable behaviours with no reward at all — for instance by rewarding skills that lead to easily distinguishable outcomes — so that the agent ends up with a toolbox of distinct movements (walking, jumping, turning) it can later sequence toward real goals.

Active learning: asking the best questions

The same "spend supervision wisely" instinct appears in supervised settings as active learning. When labels are expensive (a doctor must annotate each scan), the model itself chooses which unlabelled points to query, aiming to learn the most per label. Common selection strategies:

• Uncertainty sampling. Query the points the model is least confident about — e.g. examples nearest the decision boundary, where one label resolves the most ambiguity.
• Query-by-committee. Train several models and query the points they disagree on most.
• Expected model change / information gain. Query the point expected to shift the model the most, or reduce its uncertainty the most.

Notice the family resemblance to intrinsic motivation: both reward informativeness — novelty and surprise in exploration, uncertainty and disagreement in labelling.

Honest challenges

Letting an agent direct its own learning is powerful but genuinely hard. The main pitfalls:

Key Takeaways

• Autonomous learning is about an agent choosing what to learn and collecting its own experience with minimal supervision — the opposite of being handed a fixed, labelled dataset.
• Intrinsic motivation manufactures an internal reward for novelty or surprise (count-based bonuses like 1/√(N(s)), or prediction-error curiosity / ICM) so the agent explores even when extrinsic reward is sparse.
• As the demo shows, curiosity-driven exploration achieves broad coverage far faster than a random walk, especially through bottlenecks.
• Automatic curricula (goal generation, POET) and skill discovery (Voyager's growing skill library) extend the same frontier-seeking idea from where to look to which task to attempt and which abilities to keep.
• Active learning applies the same "maximise informativeness" principle to labelling: query the most uncertain or most-disagreed-upon points.
• The hard parts are real: defining good intrinsic rewards (noisy-TV), safe exploration, and evaluation without a fixed objective.