Introduction to Generative Models

What Makes a Model Generative?

Generative models learn to create new data that resembles their training data. Unlike discriminative models that learn boundaries between classes, generative models learn the underlying data distribution itself, enabling them to generate novel samples.

The Fundamental Goal

Given samples from an unknown distribution p_data(x), learn a model distribution p_model(x) that approximates p_data as closely as possible. Once learned, we can sample from p_model to generate new data.

Generative vs Discriminative Models

Discriminative models learn boundaries between classes, while generative models learn the full data distribution and can create new samples.

Discriminative Models:

  • • Learn decision boundaries directly
  • • Typically easier to train
  • • Better classification accuracy
  • • Cannot generate new data

Types of Generative Models

1. Explicit Density Models

Directly model p(x) and can compute likelihood of data.

  • Autoregressive: Model p(x) as product of conditionals (PixelRNN, GPT)
  • Normalizing Flows: Invertible transformations with tractable Jacobian

2. Implicit Density Models

Learn to generate samples without explicitly modeling p(x).

  • GANs: Adversarial training between generator and discriminator
  • Diffusion Models: Learn to reverse a noising process

3. Approximate Density Models

Use variational methods to approximate intractable distributions.

  • VAEs: Encode to latent space, decode to reconstruct
  • Energy-Based: Define unnormalized probability via energy function

Learning a Distribution

Watch as a generative model learns to approximate the true data distribution. The KL divergence measures how different the distributions are.

Training Process:

  • • Model starts with uniform distribution
  • • Gradually learns the true distribution shape
  • • KL divergence decreases as distributions align
  • • Generated samples become more realistic

Key Challenges

Mode Collapse

Model captures only part of the data distribution, missing diversity.

Quality vs Diversity Trade-off

High quality often comes at the cost of reduced variation.

Evaluation Metrics

How do we measure if generated samples are "good"?

  • Inception Score (IS): Measures quality and diversity
  • FID Score: Compares feature distributions
  • Perplexity: For language models
  • Human Evaluation: Still the gold standard

Evolution of Generative Models

The rapid evolution of generative models, from early energy-based models to modern transformers and diffusion models.

Key Breakthroughs:

  • • 2014: GANs introduce adversarial training
  • • 2018: Transformers revolutionize sequence modeling
  • • 2020: Diffusion models achieve state-of-the-art image quality
  • • 2023: Multimodal models blur boundaries between modalities

Applications Across Domains

Creative Applications

  • • Art & Design Generation
  • • Music Composition
  • • Creative Writing
  • • Game Content Generation

Scientific Applications

  • • Drug Discovery
  • • Protein Design
  • • Material Science
  • • Climate Modeling

Practical Applications

  • • Data Augmentation
  • • Anomaly Detection
  • • Super Resolution
  • • Compression

Communication

  • • Language Translation
  • • Code Generation
  • • Chatbots & Assistants
  • • Content Summarization

Why Generative Models Matter

  • Understanding: To generate, you must understand the underlying structure
  • Creativity: Enable machines to create, not just classify
  • Data Efficiency: Generate training data for other tasks
  • Representation Learning: Learn meaningful latent representations
  • Scientific Discovery: Explore spaces too large for exhaustive search