Text-to-Image Generation
What is the task?
Text-to-image generation takes a natural-language prompt — “a corgi astronaut floating in space, oil painting” — and produces an image that is faithful to that description. “Faithful” means two things at once: the image should be high quality and realistic (a plausible photo or painting), and it should actually contain what the prompt asked for — the right objects, the right attributes, the right relationships. Those two goals do not always agree, and balancing them is the heart of the field.
Why it is hard. The model must (1) understand open-ended language, (2) imagine a coherent scene that satisfies it, and (3) render that scene in pixels. A weakness in any stage shows up as a wrong, blurry, or off-prompt image.
The evolution: three families of approaches
Modern text-to-image grew out of three distinct modeling ideas. Each made the previous one look limited; today one family clearly dominates.
1. GAN-based (early, ~2016–2018)
A generator network maps text + noise to an image; a discriminator tries to tell real from fake. StackGAN generated coarse-to-fine in stages (a low-res sketch, then refined detail); AttnGAN added attention so individual words could influence the matching image regions. These worked on narrow domains (birds, flowers) but were hard to train (mode collapse, instability) and did not scale to open-ended prompts.
2. Autoregressive token models (DALL·E 1, 2021)
Treat an image as a sequence of discrete tokens. A VQ-VAE (vector- quantized autoencoder) compresses an image into a grid of codebook indices — so a picture becomes, say, 32×32 = 1024 discrete “image tokens.” Then a large autoregressive Transformer is trained on the concatenated sequence [text tokens, image tokens] and learns to predict each next token. At generation time you feed the text tokens and let the Transformer sample the image tokens one by one, then the VQ-VAE decoder turns them back into pixels. This was the first system to handle truly open-ended prompts.
3. Diffusion-based (current standard, 2022→)
Start from pure noise and iteratively denoise toward an image, with each step conditioned on the text. DALL·E 2 (unCLIP), Imagen, and Stable Diffusion all use this recipe and produce the sharpest, most controllable results to date. Diffusion trains stably (a simple regression loss, no adversarial game), scales well, and supports tricks like classifier-free guidance and negative prompts. Diffusion + a strong text encoder is the modern default.
The mechanics of the denoising process are covered in the Diffusion Models lesson — here we focus on the task of going from text to image.
The text encoder does the heavy lifting
Whatever the image generator, it must be conditioned on a representation of the prompt. That representation comes from a text encoder, and its quality largely determines whether the model “understood” you. Two common choices:
- CLIP text encoder — trained with a contrastive objective to put matching image/caption pairs near each other in a shared embedding space. Because it was trained on images and text together, its embeddings are already “visually grounded.” DALL·E 2 and Stable Diffusion condition on CLIP text features.
- T5 (a large language-only Transformer) — a pure text model with deep language understanding. Imagen uses a frozen T5-XXL encoder.
Key empirical finding (Imagen). Scaling up the text encoder improved prompt fidelity more than scaling up the image (diffusion) model itself. In other words, a lot of the quality of text-to-image comes from how well the language is understood before any pixel is drawn. Language understanding is a bottleneck.
Mechanically, the encoded prompt is injected into the generator via cross-attention: at each layer the image features attend to the sequence of text-token embeddings, so the words steer which regions get which content — the same attention idea AttnGAN reached for, now at scale.
Interactive demo: prompt → image, and where binding breaks
Build a prompt from structured tokens and watch a synthetic image render from them, along with a CLIP-style alignment score (the fraction of your prompt's elements the image satisfies). Then switch to two objects and try a tricky binding such as “a red cat and a blue house.” Re-sample a few times: sometimes the model swaps the colors between the objects. The picture still looks good — but it no longer matches the prompt, so alignment drops. That is attribute mis-binding, a real failure mode.
- ✓subject
- ✓color
- ✓style
- ✓composition
This simulator does not run a real generative model — it draws SVG shapes deterministically from your prompt tokens. Its point is to make the link prompt token → conditioned content visible, and to show why compositional prompts (binding the right attribute to the right object) are a real, open challenge.
Prompt adherence & compositionality: the open problem
Even state-of-the-art models can render a beautiful image that ignores part of the prompt. The hardest cases are compositional — they require reasoning about how multiple pieces fit together rather than just summoning a single concept:
- Attribute binding — “a red cube on a blue sphere.” The model may color both objects the same, or swap the colors. Binding the right attribute to the right object is hard because the conditioning signal is a bag of words that does not strongly enforce who-gets-what.
- Counting — “exactly five apples” often yields four or six.
- Spatial relations — “a spoon to the left of a cup” or “A on top of B” are frequently reversed.
- Negation — “a room with no elephant” can summon an elephant.
- Text rendering — legible words inside the image (a sign reading “OPEN”) were long a notorious weakness; character-aware text encoders and larger models have improved this but it is still imperfect.
These failures persist partly because training captions are noisy and underspecified, and partly because the loss rewards realistic images more than precisely correct ones. Improving captions (e.g. re-captioning training data with a vision-language model) is one of the most effective fixes.
How do we evaluate it?
Because “faithful” has two parts — quality and alignment — evaluation needs at least two kinds of metric, plus human judgment.
| Metric | Measures | How / caveat |
|---|---|---|
| FID (Fréchet Inception Distance) | Image quality / realism | Distance between feature distributions of generated vs. real images. Lower is better. Says nothing about whether the image matches the prompt. |
| CLIP score | Text–image alignment | Cosine similarity between the CLIP embedding of the prompt and of the generated image. Higher is better. Coarse — weak at counting and exact spatial relations. |
| Human evaluation | Both, plus subjective taste | Side-by-side preference for fidelity and alignment. Gold standard but slow and costly. |
| Compositional benchmarks | Binding, counting, relations | Targeted prompt suites (e.g. DrawBench-style / compositionality sets) probing specific skills. |
The two demo numbers map to this table: the SVG render stands in for FID-style quality(it always looks clean), while the bar is a CLIP-style alignment score. A mis-bound image can score high on the first and low on the second — exactly why you need both.
Prompt engineering & negative prompts
Because the text encoder is the steering wheel, how you phrase the prompt measurably changes the output. Practical levers:
- Be specific and front-load important tokens — name the subject, medium, lighting, and style (“studio portrait, 50mm, soft light”). Vague prompts give the model too much freedom.
- Negative prompts — many diffusion systems let you list what to avoid(“blurry, extra fingers, watermark”). Under classifier-free guidance, the sampler is pushed away from the negative prompt's direction and toward the positive one.
- Guidance scale — turning up classifier-free guidance increases adherence to the prompt, but too high over-saturates and reduces diversity. It is an alignment-vs-quality dial.
- Decompose tricky compositions — for hard binding/spatial prompts, techniques like regional prompting or layout conditioning give the model structure it cannot reliably infer from words alone.
Prompt engineering is partly a workaround for the compositionality gaps above: you are supplying, by hand, the structure the model struggles to extract from free-form text on its own.
Comparing the three families
| Family | How it generates | Strengths | Weaknesses |
|---|---|---|---|
| GAN (StackGAN, AttnGAN) | One forward pass: generator vs. discriminator | Fast sampling; sharp on narrow domains | Unstable training, mode collapse; did not scale to open prompts |
| Autoregressive (DALL·E 1) | VQ-VAE tokens, Transformer predicts them one by one | First open-ended prompts; reuses the LM toolbox | Slow sequential sampling; quantization limits detail |
| Diffusion (DALL·E 2, Imagen, SD) | Iterative denoising from noise, text-conditioned | Best quality & control; stable training; guidance / negative prompts | Many denoising steps (slower than GAN); still has compositionality gaps |
Note: the families are not mutually exclusive. Stable Diffusion runs diffusion in a compressed latent space produced by a VAE (cheaper than pixel space), and newer systems revisit autoregressive token prediction at larger scale. The clean takeaway remains: diffusion conditioned on a strong text encoder is today's standard.
Key Takeaways
- The task is to generate an image faithful to a text prompt — both realistic and matching what was asked.
- Three families evolved in turn: GAN (StackGAN, AttnGAN) → autoregressive token models (DALL·E 1: VQ-VAE discrete tokens + a Transformer) → diffusion (DALL·E 2/unCLIP, Imagen, Stable Diffusion), which now dominates.
- A strong text encoder (CLIP or T5) drives prompt understanding; Imagen found that scaling the text encoder helps more than scaling the image model.
- The prompt is injected via cross-attention, letting individual words steer image regions.
- Compositionality — attribute binding (“a red cube on a blue sphere”), counting, spatial relations, negation, and text rendering — remains the hardest open challenge.
- Evaluate with FID (quality, lower better), CLIP score (alignment, higher better), and human eval — no single metric captures faithfulness.
- Prompt engineering, negative prompts, and guidance scale are practical levers that trade off adherence against quality and diversity.
- Current standard: diffusion + a strong text encoder.