Question Answering Systems
Introduction
Question answering (QA) is the task of producing an answer to a natural-language question. It is one of the oldest goals in NLP and a benchmark for “understanding”: a system that can read a document and answer questions about it has, in some operational sense, comprehended it.
Modern QA spans a spectrum. Some systems pinpoint an exact answer inside a given passage. Others search a large corpus first, then read the retrieved documents. Still others generate an answer entirely from knowledge baked into their parameters. The right design depends on whether the answer is guaranteed to be present in some text and on whether you want a copied span or a freely-worded sentence.
The key distinction: does the system extract the answer (copy a span of text it was given) or generate it (write the answer in its own words)? This single choice shapes the architecture, the training data, and how you evaluate the result.
The Four Flavors of QA
Most QA systems fall into one of these categories. They are not mutually exclusive — an open-domain system, for example, usually has an extractive reader at its core.
Extractive QA
The answer is a contiguous span of text inside a supplied passage, and the model's job is to find it. The classic benchmark is SQuAD (Stanford Question Answering Dataset): given a Wikipedia paragraph and a question, mark the start and end of the answer. Because the answer is always present, the model never has to invent anything — it just locates.
Open-domain QA
No passage is supplied. The question could be about anything, so the system must first retrieve relevant documents from a large corpus (e.g. all of Wikipedia), then read them to extract or generate the answer. This is the retriever + reader pipeline, discussed below.
Closed-book QA
The model answers from its own parameters alone — no retrieval, no passage. The facts must have been absorbed during pre-training. This tests how much world knowledge a model has memorized. It is powerful but prone to hallucination when the parameters do not actually contain the fact.
Generative / abstractive QA
Instead of copying a span, the model writes the answer, possibly rephrasing, summarizing, or combining information from multiple sources. A sequence-to-sequence or decoder model produces the answer token by token. This is what most chat-style assistants do.
Extractive QA: Predicting a Span
Extractive QA is framed as a clean classification problem over token positions. Given a passage tokenized into positions 0, 1, 2, …, n−1, the model predicts two things:
Start position
A probability distribution over all tokens — which token does the answer begin at? It is a softmax over start logits s₀, s₁, …, sₙ₋₁.
End position
A second distribution — which token does the answer end at? A softmax over end logits e₀, e₁, …, eₙ₋₁.
The predicted answer is the span (i, j) that maximizes the combined score, subject to the sane constraint that the answer cannot end before it begins:
Here si is the start logit for token i and ej is the end logit for token j. The constraint i ≤ j guarantees a valid span. At training time the loss is the sum of two cross-entropy terms — one for the true start index, one for the true end index. Many datasets also allow a “no answer” option (SQuAD 2.0), usually handled by letting the span point at a special [CLS] token.
Why two independent distributions? It is far cheaper than scoring every one of the O(n²) possible spans directly. Predicting start and end separately turns span selection into two simple n-way classifications, and the argmax over sᵢ + eⱼ recombines them.
Reading Comprehension with BERT
BERT made extractive QA almost embarrassingly simple. You feed the question and passage together as a single sequence, separated by a [SEP] token:
BERT (an encoder-only Transformer) produces a contextual embedding for every token. To turn this into a span predictor you add just two learned vectors: a start vector S and an end vector E. The start logit for passage token i with embedding Tᵢ is their dot product:
end logiti = E · Ti
P(start = i) = softmax(S · T)i
That is the entire QA “head” — two vectors. All the heavy lifting (figuring out which passage token answers the question) is done by self-attention inside BERT, which lets every question token attend to every passage token and vice versa. The interactive demo below mimics this exact output: a start index, an end index, and a confidence.
Interactive: Extractive Span Finder
Pick a question and the system highlights the answer span it predicts inside the passage. It works by tokenizing the passage, scoring every candidate span with a transparent lexical heuristic (overlap with the question plus question-type cues like “who” → capitalized name, “when” → number/date), and reporting the best span's start and end token indices with a confidence. This is exactly the output shape of a real extractive model — only the scoring is hand-written instead of learned.
The highlighted text is the model's predicted answer span. Switch questions and watch the span move. Each token has an index; the model emits a start and end index just like a real extractive QA model.
Span length cap
Models also bound the span: end ≥ start and end − start < max.
Predicted answer
How the score works
- Reward question words appearing near the span
- Penalize a span that just echoes the question
- “Who”/“Where” → prefer capitalized names
- “When” → prefer numbers / dates
- Shorter spans preferred (length prior)
A real model learns these patterns as weights and outputs start/end logits — here the rules are hand-written so you can see every term.
Open-Domain QA: Retrieve-then-Read
When no passage is handed to you, the challenge is finding the right text before you can read it. The dominant design is a two-stage retrieve-then-read pipeline:
1. Retriever
Search a large corpus for the handful of passages most likely to contain the answer. Classic retrievers use sparse term matching (BM25); neural dense retrievers (e.g. DPR) embed the question and every passage into vectors and retrieve by nearest-neighbor search, capturing meaning rather than exact words.
2. Reader
Run an extractive (or generative) QA model over each retrieved passage and pick the best answer. This is exactly the span-prediction model from the sections above, now applied to retrieved text instead of a given paragraph.
Connection to RAG. Retrieval-Augmented Generation (RAG) is the modern, generative descendant of retrieve-then-read. Instead of an extractive reader, the retrieved passages are fed as context into a generative language model, which writes the answer while grounding it in the retrieved text. RAG reduces hallucination (the answer is conditioned on real documents) and lets you update the model's knowledge by changing the corpus rather than retraining — it is closed-book's fluency with open-domain's grounding.
Evaluating QA
For extractive QA, two metrics dominate. Both compare the predicted answer string against one or more human reference answers, after normalization (lowercasing, removing articles and punctuation).
| Metric | What it measures | Behavior |
|---|---|---|
| Exact Match (EM) | 1 if the prediction is character-for-character identical to a reference (after normalization), else 0. | Strict and all-or-nothing. “the Eiffel Tower” vs “Eiffel Tower” scores 0 unless normalized away. |
| Token-level F1 | Harmonic mean of precision and recall over the bag of shared tokens between prediction and reference. | Gives partial credit for overlapping answers, so it is more forgiving than EM. |
The F1 score treats both answers as sets of tokens. Let overlap be the number of shared tokens:
recall = overlap / (# tokens in reference)
F1 = 2 · (precision · recall) / (precision + recall)
Worked example. Reference = “Google in 2017” (3 tokens), prediction = “at Google in 2017” (4 tokens). Shared tokens = {google, in, 2017} → overlap = 3. precision = 3/4 = 0.75, recall = 3/3 = 1.0, F1 = 2·(0.75·1.0)/(0.75+1.0) ≈ 0.857. EM here would be 0 (not identical), but F1 rewards the strong overlap. Scores are computed per question (taking the max over reference answers) and averaged across the dataset.
Comparing the Approaches
| Type | Needs a passage? | Answer form | Main risk |
|---|---|---|---|
| Extractive | Yes (given) | Copied span | Answer must literally be present |
| Open-domain | No (retrieved) | Span or generated | Retrieval misses the right document |
| Closed-book | No | Generated | Hallucination — facts may be wrong |
| Generative / RAG | Optional (retrieved) | Generated | Fluent but ungrounded if retrieval is weak |
Key Takeaways
- QA splits into extractive (find a span), open-domain (retrieve then read), closed-book (answer from parameters), and generative/abstractive (write the answer).
- Extractive QA is framed as predicting a start token index and an end token index over the passage; the answer is the span (î, ĵ) = argmax over i ≤ j of (sᵢ + eⱼ).
- BERT does reading comprehension by encoding [CLS] question [SEP] passage [SEP] and adding just two learned vectors (start and end) whose dot products with token embeddings give the start/end logits.
- Open-domain QA uses a retriever (BM25 or dense/DPR) followed by a reader; RAG is its generative successor and the foundation of grounded LLM answering.
- Exact Match scores a prediction 1 only if it exactly matches a reference; token-level F1 gives partial credit via the harmonic mean of precision and recall over shared tokens.
- Choose extractive when the answer is guaranteed present and you want it copied verbatim; choose generative/RAG when you need synthesis, rephrasing, or open-ended answers — at the cost of possible hallucination.