RAG Architecture

What is RAG?

Retrieval-Augmented Generation (RAG) combines the power of large language models with external knowledge retrieval. Instead of relying solely on parametric knowledge learned during training, RAG systems dynamically fetch relevant information to generate more accurate, up-to-date, and verifiable responses.

The key mechanism is vector similarity, not keyword matching. Every chunk of your source corpus is embedded into a high-dimensional vector and stored. When a question arrives, the query itself is embedded into that same vector space using the same embedding model. Retrieval then becomes geometry: the system computes a distance (or similarity) between the query vector and every chunk vector — typically cosine similarity or a dot product — and keeps the top-N nearest chunks. Those chunks, together with their metadata (chunk ID, source document, position) and the original question, are what gets handed to the LLM so it can write a grounded answer. Because matching happens on meaning rather than exact words, a query like “how do I reset my password” can retrieve a passage titled “account recovery steps” even though they share no keywords.

Why RAG is Needed

LLM Limitations

  • Knowledge cutoff date
  • Can't access private data
  • May hallucinate facts
  • Can't cite sources

RAG Solutions

  • Real-time information access
  • Query private knowledge bases
  • Grounded in retrieved facts
  • Provides source attribution

RAG Pipeline Architecture

Core Components

1. Indexing the corpus (offline, done once)

Documents → Chunks

Split source into passages

Embeddings

Each chunk → a vector

Vector DB

Vectors + metadata stored

2. Answering a query (online, every question)

User Query

The question

Embed Query

Same vector space as chunks

Similarity Search

Cosine / dot product vs. all chunks

Top-N Chunks

Nearest matches + metadata

LLM

Chunks + metadata + query

Grounded Answer

With citations

How retrieval actually works: the query is vectorized with the same embedding model used for the corpus, so the question and the chunks live in one shared space. The retriever then computes a distance/similarity score (cosine similarity, dot product, or Euclidean distance) between the query vector and every stored chunk vector, ranks them, and selects the top-N nearest chunks.

What gets passed to the model is not just the raw text: each retrieved chunk travels with its identifying metadata — chunk ID, source document, and position — alongside the original query. That bundle is assembled into the prompt so the LLM can ground its answer in the passages and cite exactly where each fact came from.

Step-by-Step Process

  1. 1
    Document Processing: Split documents into chunks, clean text, extract metadata
  2. 2
    Embedding Generation: Convert text chunks into high-dimensional vectors
  3. 3
    Indexing: Store embeddings in vector database with efficient search structures
  4. 4
    Query Vectorization: When a question arrives, embed it with the same embedding model so the query vector lands in the same space as the chunk vectors
  5. 5
    Distance / Similarity Calculation: Score the query vector against every stored chunk vector using cosine similarity, dot product, or Euclidean distance
  6. 6
    Top-N Selection: Rank chunks by score and keep the N nearest matches, carrying their metadata (chunk ID, source, position) along with them
  7. 7
    Context Assembly: Combine the retrieved chunks and their metadata with the original query into a single prompt
  8. 8
    Response Generation: The LLM generates a grounded answer using the retrieved context, able to cite which source each fact came from

Architecture Patterns

Basic RAG

Simple retrieval and generation

Components

  • • Embedding Model
  • • Vector DB
  • • LLM
  • • Simple Prompt

Pros

  • ✓ Easy to implement
  • ✓ Low complexity
  • ✓ Quick prototyping

Cons

  • ✗ Limited accuracy
  • ✗ No query optimization
  • ✗ Basic context handling

RAG Implementation Libraries

LangChain

Python/JS
Best for: Full-stack RAG applications

Key Features:

Extensive integrationsChain abstractionsMemory management
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chains import RetrievalQA

# Initialize vector store
vectorstore = Chroma.from_documents(
    documents=docs,
    embedding=OpenAIEmbeddings()
)

# Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    return_source_documents=True
)

LlamaIndex

Python
Best for: Document-heavy applications

Key Features:

Data connectorsIndex structuresQuery engines
from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms import OpenAI

# Load and index documents
documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents)

# Query with RAG
query_engine = index.as_query_engine(
    llm=OpenAI(model="gpt-4"),
    similarity_top_k=3
)
response = query_engine.query("What is RAG?")

Haystack

Python
Best for: Enterprise search systems

Key Features:

Pipeline architectureProduction-readyMulti-modal support
from haystack import Pipeline
from haystack.nodes import EmbeddingRetriever, PromptNode

# Build RAG pipeline
pipeline = Pipeline()
pipeline.add_node(
    component=retriever,
    name="Retriever",
    inputs=["Query"]
)
pipeline.add_node(
    component=prompt_node,
    name="PromptNode",
    inputs=["Retriever"]
)

Production Challenges

Latency

RAG adds retrieval time to generation

Solutions:

  • Cache frequent queries
  • Optimize embedding dimensions
  • Use faster vector indexes
  • Parallel retrieval and generation

Context Window Limits

LLMs have token limits for context

Solutions:

  • Implement context compression
  • Use hierarchical summarization
  • Smart chunk selection
  • Sliding window approach

Retrieval Quality

Retrieved chunks may not be relevant

Solutions:

  • Hybrid search (vector + keyword)
  • Query expansion/rewriting
  • Cross-encoder re-ranking
  • Feedback loops for improvement

Data Freshness

Keeping vector index up-to-date

Solutions:

  • Incremental indexing
  • Real-time embedding pipeline
  • Version control for embeddings
  • Scheduled re-indexing

Advanced RAG Patterns

Multi-Query RAG

Generate multiple query variations to improve retrieval coverage

# Generate multiple queries from user input
queries = [
    "What is transformer architecture?",
    "How do transformers work in NLP?",
    "Explain self-attention mechanism",
    "Transformer model components"
]

# Retrieve for each query and merge results
all_docs = []
for query in queries:
    docs = retriever.get_relevant_documents(query)
    all_docs.extend(docs)

# Deduplicate and rank
unique_docs = deduplicate(all_docs)
ranked_docs = rerank(unique_docs, original_query)

RAG with Guardrails

Add safety and validation layers to RAG pipeline

  • • Input Validation: Check queries for malicious content
  • • Source Verification: Ensure retrieved docs are from trusted sources
  • • Output Filtering: Remove sensitive information from responses
  • • Hallucination Detection: Verify claims against retrieved context

Adaptive RAG

Dynamically adjust retrieval strategy based on query type

Simple Queries

Direct retrieval → Generate

Complex Queries

Decompose → Multi-hop retrieval

Comparison Queries

Parallel retrieval → Synthesize

Evaluation Metrics

Measuring RAG Performance

Retrieval Metrics

  • Precision@K: Relevant docs in top K results
  • Recall@K: Coverage of all relevant docs
  • MRR: Mean Reciprocal Rank of first relevant doc
  • NDCG: Normalized Discounted Cumulative Gain

Generation Metrics

  • Faithfulness: Answer grounded in retrieved context
  • Relevance: Answer addresses the query
  • Completeness: All aspects of query covered
  • Coherence: Logical flow and clarity

Try It: Build a Retrieval Pipeline

Everything above, made concrete. Paste your own text, watch it get chunked and embedded into real vectors, see those vectors laid out in 3D by t-SNE, then run a query and watch retrieval pick out the nearest chunks. Notice that the lab embeds your query into the same vector space as the chunks — that is exactly the query-vectorization step above. It then computes the similarity between that query vector and every chunk vector and highlights the top-N nearest ones, demonstrating the same distance calculation a production retriever runs. This is the retrieval half of RAG, end to end.

Load an example:
0 / 5000 chars

Next Steps

Now that you understand RAG architecture and implementation, explore how modern online search tools like Perplexity combine RAG with web-scale search capabilities.

Continue to Online LLM Search