Advanced RAG Techniques

Beyond retrieve-then-read

Naive RAG is a straight line: embed the user's question, grab the top-k nearest chunks by vector similarity, paste them into the prompt, and generate. It works surprisingly often — and fails in surprisingly predictable ways. The question may be phrased nothing like the answer. The chunks may be the wrong size. Vector similarity may rank a passage highly just because it shares vocabulary, not because it answers anything. And the model has no way to notice that the retrieved context is junk before it confidently writes from it.

Advanced RAG is the collection of techniques that intervene at each of these failure points. They cluster into stages of the pipeline:

1. Query transformation: fix the question before you search

The user's raw question is often a bad search key. It may be terse, ambiguous, multi-part, or phrased in language that looks nothing like the documents that answer it. Query transformation rewrites the question into one or more better retrieval keys before embedding.

query   = "What causes the northern lights?"
hypo    = llm("Write a passage answering: " + query)
#         -> "The aurora is caused by charged solar
#             particles exciting oxygen and nitrogen..."
results = vector_search(embed(hypo))   # embed the ANSWER, not the query

2. Better chunking: retrieval quality starts at index time

You cannot retrieve well from badly cut chunks. Fixed-size chunking (e.g. every 512 tokens) routinely slices a sentence — or a key fact — in half, leaving each piece too incomplete to embed meaningfully. Smarter chunking respects the structure of the text.

3. Re-ranking: the two-stage retrieve → rerank pattern

This is the single highest-leverage upgrade to naive RAG, and the centrepiece of this lesson. The idea: use a cheap, recall-oriented retriever to pull a broad candidate set, then a more expensive, precision-oriented reranker to reorder just those candidates. The split exists because of a fundamental tradeoff between two model architectures.

Interactive: retrieve → rerank

A fixed corpus of 12 short passages with toy precomputed vectors. Stage 1 scores the whole corpus by cosine similarity (the bi-encoder). Stage 2 reranks only the top-N candidates with a query-aware cross-encoder score. Pick a query, adjust N, and watch the ranking change — and the precision improve.

4. Hybrid & fusion retrieval

Dense vector search captures meaning but can miss exact terms — product codes, rare names, acronyms. Sparse keyword search (BM25) nails exact terms but misses paraphrases. Hybrid retrieval runs both and combines them, so you get semantic recall and lexical precision together.

5. Contextual compression & self-corrective RAG

6. Evaluating a RAG system

You cannot improve what you cannot measure, and RAG has two halves to measure separately: did retrieval find the right evidence, and did generation use it faithfully? Frameworks like RAGAS compute these, often using an LLM as a judge so you do not need human labels for every example.

Putting it together

A mature RAG pipeline threads these techniques together — though you should add each one only when evaluation shows it earns its latency and cost:

Key Takeaways

• Naive retrieve-then-read fails at predictable points; advanced RAG adds targeted fixes at the pre-retrieval, retrieval, post-retrieval, and self-correction stages.
• Query transformation repairs the search key before embedding: rewriting and multi-query expansion improve recall, HyDE embeds a hypothetical answer to bridge the question/answer gap, and step-back prompting retrieves at a higher level of abstraction.
• Chunking determines the ceiling on retrieval quality. Recursive and semantic chunking respect text structure; sentence-window and parent-document retrieval decouple the small unit you match from the larger unit you read.
• The retrieve → rerank pattern is the highest-leverage upgrade: a cheap bi-encoder (separate, precomputable vectors) recalls a broad candidate set, then an accurate cross-encoder (query and doc encoded jointly, not precomputable) reorders just those candidates for precision.
• Hybrid retrieval combines dense and sparse search; Reciprocal Rank Fusion merges ranked lists using only ranks — RRF(d) = Σ 1/(k + rankᵢ(d)) — so incomparable score scales never need calibrating.
• Contextual compression trims retrieved chunks to the relevant sentences; Self-RAG and CRAG add a grade-and-correct loop that re-retrieves or falls back when the context is poor.
• Measure retrieval and generation separately: context recall/precision diagnose retrieval, and faithfulness diagnoses grounding. RAGAS automates these with an LLM judge, telling you which stage to improve.