Knowledge Graph RAG
Why pure vector RAG hits a wall
Standard RAG embeds your documents into chunks, stores the vectors, and at query time retrieves the top-k chunks whose embeddings are closest to the question. This is excellent when the answer lives inside a single passage that is locally similar to the query. It struggles badly when the answer is spread across documents or requires connecting facts.
Three failure modes show up again and again:
- Multi-hop questions. "Who founded the company that makes Engine X?" needs two linked facts — the maker of Engine X, and the founder of that maker. The chunk that mentions Engine X usually does not mention the founder, so vector search retrieves a relevant-looking chunk that simply does not contain the answer.
- Global / structural questions. "What are the main themes across this whole corpus?" has no single similar chunk to retrieve — the answer requires synthesizing structure over the entire dataset, not finding one nearest neighbor.
- Aggregation across documents. "How many products does each company make?" requires gathering and counting facts that are scattered across many chunks. Similarity ranking cannot aggregate.
The root cause: embeddings capture local semantic similarity, but they do not encode the explicit relationships between entities. Knowledge Graph RAG (commonly "GraphRAG") adds those relationships back as first-class structure you can traverse.
Knowledge graphs: entities and typed relationships
A knowledge graph represents information as a network. Entities are the nodes (people, companies, products, places, concepts). Relationships are the edges, and crucially they are typed — each edge carries a label describing how the two entities are connected.
The atomic unit of a knowledge graph is the triple, written as (subject, predicate, object) — also called an SPO triple. Each triple is one labeled edge:
Chain triples that share an entity and you get a path. The first two triples above share Analytica Inc., so they connect Ada Lovelace to Engine X through two hops. That chained path is exactly what a multi-hop question needs — and exactly what a single similar chunk cannot give you.
Key shift: vector RAG retrieves by similarity (which chunk looks most like the query). Graph RAG retrieves by connectivity (which entities are linked to the entities in the query, and how). Similarity finds a neighborhood in embedding space; traversal walks explicit relationships in the graph.
Interactive demo: multi-hop graph traversal
Below is a small fixed knowledge graph of people, companies, products, and a place. Pick a multi-hop question, then step through retrieval. In Graph traversal mode, retrieval grounds the question on a start entity, expands to 1-hop neighbors, then 2-hop, collecting the triples along the path — the assembled context that would be fed to the LLM. Flip to Vector RAG mode to see what a single top-1 similar chunk retrieves instead, and why it misses the connection.
Hop 0 of 2. Click “Start at entity” to ground the question on a node, then expand neighbors hop by hop.
The graph is a toy with 7 entities; real GraphRAG systems traverse graphs with millions of nodes, often bounding the search to a 1–2 hop neighborhood around the entities mentioned in the query to keep retrieval fast.
How GraphRAG works end to end
GraphRAG has two phases: an offline indexing phase that builds the graph from your corpus, and an online retrieval phase that queries it.
1. Graph construction (indexing)
- 1Chunk the corpus as in normal RAG.
- 2Extract entities and relations from each chunk, most often with an LLM prompted to emit (subject, predicate, object) triples (this is LLM-assisted information extraction). Classical NER plus relation-extraction models can do the same job.
- 3Merge into a graph. The same entity mentioned in many chunks (e.g. resolving "Analytica" and "Analytica Inc." to one node) becomes a single node, so facts from different documents get connected. This merging is what enables cross-document reasoning.
- 4(Optional) Detect communities and summarize. Run a graph clustering algorithm (e.g. Leiden) to group densely-connected entities into communities, then have an LLM write a summary of each community. These summaries power global queries (see below).
2. Retrieval
There are two complementary retrieval strategies, suited to different question types:
Local: graph traversal
Ground the query on the entities it mentions, then traverse the graph to collect a multi-hop neighborhood / path around them. The retrieved context is the set of triples (and their source chunks) along the way. This is what answers specific multi-hop questions like the demo above.
Global: community summaries
For broad "what are the main themes?" questions, retrieve the pre-computed community summaries, have the LLM answer from each, then combine those partial answers (a map-reduce over the graph). This is the core idea of Microsoft's GraphRAG for global sensemaking.
Microsoft GraphRAG popularized this pattern: extract a graph with an LLM, cluster it into a hierarchy of communities, summarize each community, and answer global queries by map-reducing over the community summaries — substantially outperforming naive vector RAG on corpus-wide questions where there is no single chunk to retrieve.
Hybrid graph + vector retrieval
In practice the two approaches are complementary, not competing. A common hybrid pipeline:
- Vector search finds the chunks (and thus the entities) most semantically relevant to the query — a fast way to ground the query onto entry-point nodes in the graph.
- Graph traversal then expands outward from those entry points to pull in connected facts that vector search alone would miss.
- The combined context — similar chunks plus the relationship path — is handed to the LLM, which often also gets the explicit path so it can cite its reasoning.
Vector search supplies recall and grounding; the graph supplies connection and structure. Using both gives you the semantic fuzziness of embeddings and the precise multi-hop reach of traversal.
Benefits versus costs
| Dimension | Pure vector RAG | Knowledge Graph RAG |
|---|---|---|
| Multi-hop reasoning | Weak — answer often not in the single similar chunk | Strong — chains triples along an explicit path |
| Global / corpus-wide questions | Weak — no single nearest chunk to retrieve | Strong — community summaries synthesize structure |
| Explainability | Cites retrieved chunks | Cites explicit relationship paths (which entity, via which edge) |
| Indexing cost | Low — embed and store chunks | High — LLM-assisted extraction, entity resolution, clustering, summarization |
| Maintenance | Re-embed changed chunks | Re-extract and re-merge; keep the graph consistent |
| Best for | Single-fact lookups, semantic search | Multi-hop, aggregation, global synthesis |
Benefits
- Multi-hop reasoning across linked entities
- Global synthesis via community summaries
- Explainable answers — the path is the evidence
- Cross-document aggregation through merged entities
Costs
- Expensive graph construction (often many LLM calls)
- Extraction errors / noisy or wrong triples propagate
- Entity resolution is hard — duplicate or merged nodes
- More moving parts to keep fresh as the corpus changes
Key Takeaways
- Pure vector RAG retrieves locally-similar chunks; it is weak at multi-hop reasoning, global/structural questions, and cross-document aggregation because embeddings encode similarity, not explicit relationships.
- A knowledge graph stores entities as nodes and typed relationships as edges; its atomic unit is the (subject, predicate, object) triple, and chained triples form multi-hop paths.
- GraphRAG extracts a graph from the corpus (often LLM-assisted), then retrieves by traversing multi-hop neighborhoods and/or by using community summaries for global queries (the Microsoft GraphRAG pattern: cluster, summarize, map-reduce).
- Hybrid retrieval is the practical default: vector search grounds the query onto entry-point entities, and graph traversal expands to the connected facts vector search misses.
- The trade-off is capability vs. cost: graphs unlock multi-hop reasoning, global synthesis, and explainable paths, but pay for it with heavy construction, extraction noise, and entity-resolution and maintenance overhead.