How LLMs Reason
Exploring the mechanisms behind reasoning in large language models
Introduction
Large Language Models (LLMs) produce outputs that look like reasoning, even though their pretraining objective is just next-token prediction (later refined with instruction tuning and reinforcement learning). This lesson explores the mechanisms behind this behavior — and why “reasoning” here is closer to learned pattern-matching than to formal logic.
Emergent Reasoning
Reasoning in LLMs emerges from pattern recognition at scale. Key phenomena include:
- In-context learning: Models adapt to new tasks from examples in the prompt
- Chain-of-thought: Step-by-step reasoning improves accuracy on complex tasks
- Few-shot reasoning: Generalizing from limited examples
- Multi-hop reasoning: Connecting disparate pieces of information
Mechanisms of Reasoning
1. Attention-Based Information Flow
Self-attention mechanisms allow models to relate different parts of the input, creating computational paths for reasoning:
# Simplified attention for reasoning Q = "What is 2+2?" K = ["2", "+", "2", "="] V = ["two", "plus", "two", "equals"] # Attention weights focus on relevant tokens # Model learns arithmetic patterns through training
2. Internal Representations
Hidden states encode abstract concepts that support reasoning:
- Early layers: Syntactic and lexical features
- Middle layers: Semantic relationships and concepts
- Later layers: Task-specific computations and reasoning
3. Compositional Reasoning
Models compose simple operations into complex reasoning chains:
| Operation | Example |
|---|---|
| Retrieval | "Paris is the capital of France" |
| Inference | "Therefore, Paris is in France" |
| Generalization | "Capitals are typically large cities" |
Limitations and Challenges
Consistency Issues
Models may produce different reasoning paths for similar problems, lacking systematic approaches.
Hallucination
Confident generation of plausible but incorrect reasoning steps, especially in knowledge gaps.
Formal Logic
Struggle with strict logical operations and mathematical proofs requiring symbolic manipulation.
Causal Understanding
Limited grasp of true causality, often relying on correlational patterns from training data.
Improving Reasoning
Recent advances in enhancing LLM reasoning capabilities:
- 1. Instruction Tuning: Fine-tuning on reasoning tasks with step-by-step explanations
- 2. Self-Consistency & Test-Time Compute: Sampling several chain-of-thought traces and taking a majority vote, or spending more tokens “thinking” before answering
- 3. Tool Use: Augmenting reasoning with calculators, search, and symbolic solvers
- 4. Reinforcement Learning: Optimizing for reasoning that reaches correct answers — via RLHF for general helpfulness, and via RL on verifiable rewards (e.g. math and code) to strengthen multi-step reasoning
Interactive Example
The classic “widgets” puzzle exposes a shortcut LLMs (and people) take: pattern-matching 5→5→5 to 100→100→100. Change the numbers and step through the reasoning to see why the correct answer depends only on the per-machine rate, not the totals.
Naive pattern-match answer
100.00 min
Scales the totals as if more machines did not help.
Step-by-step (correct) answer
5.00 min
The naive answer is wrong — parallel machines change the time.
Reasoning trace:
- (reveal the steps to follow the chain of thought)
Next Steps
Continue learning about generative AI: