Video Generation
Introduction
Video generation asks a model to synthesize a sequence of images that, played in order, look like a coherent moving scene. At first glance it sounds like "image generation, but more of it." The twist is the word more: a video is images plus time, and time is exactly what makes the problem hard.
Two consequences fall out immediately. First, the frames cannot be generated independently — they must agree with one another so that an object keeps its identity, color, and shape and moves smoothly. This is called temporal coherence, and it is the central challenge of the field. Second, a video is simply a lot more data: a 5-second clip at 24 frames per second is 120 images, so the compute and memory dwarf single-image generation.
The one idea to keep: video = images + time. The added time axis turns "make a good picture" into "make many pictures that stay consistent with each other," which is both the hard part and the reason video models cost so much to run.
Why temporal coherence is the hard part
Imagine generating each frame with a strong image model and stitching them together. Even tiny per-frame randomness compounds across a clip and shows up as recognizable failure modes:
Flicker
Colors, textures, or shapes pop and change between adjacent frames instead of staying fixed.
Identity drift
An object slowly morphs — a face changes, a logo mutates, a ball grows or shrinks — losing its identity over time.
Jitter & physics
Motion looks shaky or implausible: objects teleport, pass through each other, or move in ways that violate everyday physics.
The fix is to let frames see each other during generation, so the model can enforce consistency across time rather than hoping it emerges by luck. The interactive below lets you dial that consistency up and down and watch the failure modes appear.
Interactive: the temporal-coherence visualizer
A short synthetic "generated video" of a ball crossing the scene. Drag the temporal consistency slider, scrub the timeline, and press play. At high consistency the ball glides smoothly with a stable color and shape; at low consistency you see flicker, identity drift, and jitter — the very things a real video model must suppress.
Temporal consistency
CoherentSlide left to weaken cross-frame consistency and watch the failure modes appear.
Timeline scrubber
What to look for
- Flicker: color/shape jumps frame-to-frame
- Identity drift: the ball slowly changes size
- Jitter: position wobbles off the smooth path
- Gray dots = the intended smooth trajectory
How modern video models actually work
1. Video diffusion: image diffusion + a time axis
Most video generators build on diffusion models, the same family behind modern image generators. A diffusion model learns to reverse a noising process: start from pure noise and denoise step by step into a clean sample. To make video, you extend the model so it denoises a whole stack of frames at once, adding a temporal dimension to the network.
There are two common ways to add that dimension. A 3D U-Net replaces 2D convolutions with 3D ones that operate over height, width, and time. More popular today is factorized attention: alternate between spatial attention (patches attend within a frame, handling appearance) and temporal attention (each location attends to itself across frames, handling motion and consistency). Factorizing is a compute trick — attending to every patch in every frame jointly would be astronomically expensive.
Click any patch. In spatial mode it mixes with patches in the same frame (appearance, layout). In temporal mode it mixes with the same location across frames (motion, consistency). Real video models factorize attention this way to keep compute tractable.
2. Latent video diffusion: denoise in a compressed space
Denoising raw pixels for 120 frames is wasteful. Latent video diffusion first uses an autoencoder to compress the video into a much smaller latent representation, runs diffusion there, and then decodes back to pixels. This is the video analogue of latent image diffusion (e.g. Stable Diffusion) and is what makes longer, higher-resolution clips feasible.
3. Diffusion transformers on spacetime patches (the Sora approach)
The current frontier replaces the U-Net with a diffusion transformer (DiT). The recipe, popularized by OpenAI's Sora:
- Compress the video into a lower-dimensional spatiotemporal latent with an encoder.
- Cut that latent into "spacetime patches" — small cubes spanning a little area and a little time. Each patch becomes one token, just like image patches become tokens in a Vision Transformer.
- Run a diffusion transformer over the sequence of spacetime patch-tokens, denoising them together. Because attention spans the whole sequence, the model can reason about appearance and motion jointly — coherence is built in rather than bolted on.
Why patches matter: turning video into a flat sequence of spacetime tokens lets one transformer handle clips of different resolutions, aspect ratios, and durations — and lets it scale with more compute the same way large language models do.
4. Conditioning: what you steer it with
The same backbone is conditioned in different ways:
- Text-to-video: a text prompt (encoded by a language model) guides the denoising, as in text-to-image.
- Image-to-video / animation: a starting image is fixed and the model generates plausible motion forward from it.
- Video-to-video & extension: condition on existing frames to continue, in-fill, or restyle a clip.
Approaches at a glance
| Approach | How time is handled | Strengths |
|---|---|---|
| 3D U-Net diffusion | 3D convolutions over (H, W, T) | Direct extension of image U-Nets; strong local motion |
| Factorized spatial + temporal attention | Alternating within-frame and across-frame attention | Cheaper than full 3D attention; explicit consistency |
| Latent video diffusion | Diffusion in a compressed spatiotemporal latent | Far less compute; enables longer/HD clips |
| Diffusion transformer on spacetime patches | Latent cut into patch-tokens; transformer over the sequence | Scales with compute; flexible duration/resolution (Sora-style) |
Open challenges & how we measure quality
Still hard
- • Long-range consistency: keeping identity over many seconds
- • Physics plausibility: gravity, collisions, object permanence
- • Flicker in texture and lighting
- • Cost: many frames means heavy compute and memory
- • Control: precise, frame-accurate direction is limited
Measuring it
The most common automatic metric is FVD (Fréchet Video Distance). It extends the image metric FID to video: it compares the distribution of features (from a pretrained video network) for generated vs. real clips. Lower FVD means the generated videos are statistically closer to real ones — capturing both per-frame quality and temporal dynamics.
Because no single number fully captures "looks right," FVD is usually paired with human evaluation and prompt-alignment scores.
A note on capabilities: today's best models produce striking short clips, but they do not have a true physics engine. They learn statistical regularities from data, so they can still break object permanence or physical plausibility, especially over longer durations. Impressive is not the same as reliable.
Key Takeaways
- Video = images + time; the added time axis is what makes generation hard.
- Temporal coherence — consistent identity, color, shape, and smooth motion across frames — is the central challenge.
- Generating frames independently causes flicker, identity drift, and jitter; frames must be generated jointly so they can enforce consistency.
- Video diffusion extends image diffusion with a temporal dimension via a 3D U-Net or factorized spatial + temporal attention.
- Latent video diffusion denoises in a compressed space to make longer, higher-resolution clips feasible.
- The modern frontier (e.g. Sora) compresses video into spacetime latent patches and runs a diffusion transformer over them, building coherence in by design.
- Conditioning gives text-to-video, image-to-video/animation, and video extension from one backbone.
- Quality is measured with FVD (lower is better), plus human evaluation; current models lack a true physics engine and remain costly.