Image-to-Text Models
What are image-to-text models?
An image-to-text model takes pixels in and produces natural-language text out. It is a translation problem across two very different modalities: a grid of color values on one side, a sequence of words on the other. The model has to perceive what is in the image and then describe it in fluent language.
There are several closely related tasks under this umbrella:
Image captioning
Produce a one-line description of the whole image. “A dog is throwing a red ball on the grass.”
Visual question answering (VQA)
Answer a free-form question about an image. “What color is the ball?” → “Red.”
OCR (optical character recognition)
Read text that appears inside the image — signs, receipts, documents — and transcribe it.
Dense captioning
Detect many regions and caption each one separately — a description per object, with its bounding box.
The classic architecture: encoder → decoder
The dominant design is an encoder–decoder pair, exactly like neural machine translation, but the “source language” is an image:
- Image encoder. A CNN (e.g. a ResNet) or, more recently, a Vision Transformer (ViT) ingests the image and produces a set of feature vectors — one per spatial location or image patch. You can think of these as a grid of region descriptors.
- Language decoder. An autoregressive language model (originally an LSTM/RNN, now a Transformer decoder) generates the caption one token at a time, feeding each emitted word back in as the next input.
Show-and-Tell vs. Show-Attend-and-Tell
Show and Tell (Vinyals et al., 2015) squeezed the whole image into a single feature vector and used it to seed an LSTM. Simple, but the decoder cannot “look back” at specific parts of the image.
Show, Attend and Tell (Xu et al., 2015) added visual attention: the encoder keeps a grid of region features, and at every word the decoder computes a weighted average over those regions. The weights say where to look while emitting that word — this is exactly what the demo below visualizes.
The cross-attention that connects decoder to encoder is the same scaled dot-product attention used everywhere in Transformers. The decoder's current state forms the query Q; the image region features form the keys K and values V:
Here Q comes from the text side (what the decoder wants to say next), while K and V come from the image side (the region features). The softmax produces one weight per image region — those are the attention weights you see lighting up the grid. dₖ is the key dimension, and dividing by √dₖ keeps the dot products from growing too large.
Interactive demo: watch attention ground each word
Below is a tiny synthetic “image” — a 4×4 grid of regions, each labeled with what it contains (sky, dog, arm, grass, ball). Step through caption generation one word at a time. As each word is produced, the cells the decoder attends to glow amber, with the attention weight shown in each cell (the 16 weights always sum to 1). This is the grounding: the model produces “ball” because it is looking at the ball region. Toggle to VQA to ask “what color is the ball?” and watch attention focus before the answer comes out.
Generated caption
Function word — attention is spread out / diffuse. Not all words are grounded in a region.
What to notice
- Content words (dog, red, ball, grass) sharply focus attention on the matching region.
- Function words (a, is, the) leave attention diffuse — they are not visually grounded.
- The image encoding is fixed; only the decoder's attention moves as each word is produced.
(The attention weights here come from a fixed teaching script rather than a live network, but they faithfully illustrate the real mechanism: a normalized distribution over image regions, recomputed at every decoder time-step.)
Modern Vision-Language Models (VLMs)
Today the field has converged on a powerful recipe: take a pretrained vision encoder (usually a CLIP-style ViT) and a pretrained large language model, then connect them with a small trainable projector that maps image features into the LLM's token-embedding space. The image effectively becomes a handful of “visual tokens” that sit at the front of the LLM's context, and the LLM generates text conditioned on them.
Different systems implement the “projector” differently — this is the main design axis:
| Model | How image meets language |
|---|---|
| LLaVA | Simplest pattern: a CLIP ViT encodes the image, a small MLP projection maps the patch features into the LLM's embedding space, and they are prepended as tokens. |
| BLIP-2 | Inserts a Q-Former — a small Transformer with a fixed set of learned query tokens that extract a compact summary from the frozen vision encoder and feed it to the LLM. Keeps both big models frozen and trains only the bridge. |
| Flamingo | Uses a Perceiver Resampler to turn variable-size visual features into a fixed number of tokens, then injects them into a frozen LLM via gated cross-attention layers inserted between the LLM's blocks. Handles interleaved image-and-text input. |
| GPT-4V / Gemini | Proprietary, natively multimodal: images and text share one model. Architectural details are undisclosed, but they follow the same broad “vision encoder feeding a language model” idea, trained at very large scale. |
Two patterns, same idea. The classic encoder–decoder injects the image via dedicated cross-attention (Flamingo keeps this style). The LLaVA-style approach instead turns the image into tokens in the input sequence, so the LLM's ordinary self-attention does the cross-modal mixing. Both let every generated word condition on the image.
How they are trained
1. Pretraining on image–caption pairs. The base signal is enormous datasets of (image, text) pairs scraped from the web (alt-text, captions). For a generative model the loss is the usual next-token cross-entropy: maximize the probability of each caption token given the image and the preceding tokens. The vision encoder itself is often pretrained separately with a contrastive objective (CLIP), aligning image and text embeddings.
2. Visual instruction tuning. To make a VLM follow open-ended requests (“describe this”, “count the cars”, “is this safe?”), it is fine-tuned on multimodal instruction data: (image, question/instruction, desired answer) triples, often generated or curated with the help of a strong language model. This is what turns a captioner into a conversational visual assistant.
A common efficiency trick (BLIP-2, Flamingo, early LLaVA) is to freeze the expensive vision encoder and LLM and train only the small bridge/projector. This makes alignment cheap and preserves the LLM's language abilities.
Evaluation
Because the output is text, image-to-text models are scored differently per task:
| Metric | Task | What it measures |
|---|---|---|
| BLEU | Captioning | N-gram precision overlap with reference captions. Borrowed from machine translation; rewards matching word sequences. |
| CIDEr | Captioning | TF-IDF-weighted n-gram similarity to a set of human references — designed specifically for captioning, downweighting common words. |
| SPICE | Captioning | Parses captions into a scene graph (objects, attributes, relations) and compares those — it cares about semantic content, not just word overlap. |
| Accuracy | VQA | Fraction of questions answered correctly. The VQA benchmark counts an answer correct if enough of the human annotators gave it. |
All of these are imperfect: a caption can be perfectly accurate yet share few n-grams with the references, so n-gram metrics undervalue it. This is why semantic metrics (SPICE) and, increasingly, model-based / human evaluation are used alongside them.
Hallucination in VLMs
A central failure mode is object hallucination: the model confidently describes things that are not in the image. It might mention a “person” in a kitchen photo simply because kitchens usually contain people, or invent a color, count, or text on a sign.
Why it happens
- Language priors dominate. The LLM has strong expectations about what co-occurs; when the visual signal is weak it falls back on those priors instead of the pixels.
- Compression loss. Squeezing an image into a few visual tokens discards detail, so fine-grained facts (exact count, small text, precise color) may simply not reach the decoder.
- Weak grounding. If attention is not tightly tied to the right region, generated words drift away from what is actually there.
Mitigations include training on harder negative examples, better grounding objectives, retrieval/tool use for reading text, and decoding strategies that downweight the language prior in favor of the visual evidence (contrastive decoding). Notably, the attention grounding in the demo above is exactly the signal researchers probe to check whether a generated word is supported by the image.
Key Takeaways
- Image-to-text covers captioning, VQA, OCR, and dense captioning — all map pixels to language.
- The classic design is an image encoder (CNN/ViT) + language decoder, joined by cross-attention: softmax(QKᵀ/√dₖ)V with the query from text and keys/values from image regions.
- Visual attention (Show, Attend and Tell) lets the decoder look at different image regions for different words — this grounding is what the interactive demo visualizes.
- Modern VLMs follow vision encoder → projector → LLM: LLaVA (MLP), BLIP-2 (Q-Former), Flamingo (Perceiver + gated cross-attention), GPT-4V / Gemini (natively multimodal).
- Training = image–caption pretraining (often contrastive CLIP encoders) plus multimodal instruction tuning; freezing the big models and training only the bridge is common.
- Evaluate captioning with BLEU / CIDEr / SPICE and VQA with accuracy; all metrics are imperfect proxies.
- Hallucination — describing things that are not present — is the key reliability risk, driven by language priors, lossy compression, and weak grounding.