AI Deep-dive: GraphRAG Explained - How Knowledge Graph Retrieval Works and When It Is Worth the Cost

What GraphRAG solves that standard RAG cannot

Standard RAG fails quietly on two question types that matter in finance work, and GraphRAG was built to address both. This post explains how it works, what it costs, and how to decide whether your problem actually needs it.

The two failure modes are chained retrieval across documents and synthesis questions spanning the full corpus. GraphRAG handles them by building a knowledge graph from your text and layering a community detection structure on top. Before you commit to that approach, though, it is worth understanding exactly what you are taking on.

How standard RAG works and where it stops

The mechanics are worth stating plainly because the gap becomes obvious once you see them.

Standard RAG takes your documents, splits them into chunks, embeds each chunk as a vector, and stores those vectors in an index. At query time it embeds the question, runs an approximate nearest neighbour search to find the most similar chunks, and passes those chunks to a large language model (LLM) as context. The retrieval step is fast and the indexing is cheap: one embedding model pass over the corpus.

What it cannot do is traverse relationships. The embedding space captures semantic similarity between passages, not logical connections between entities. If the fact you need is spread across five documents and requires three reasoning steps to assemble, the retrieval step has no mechanism to discover that chain. It returns whatever chunks look most like the question.

The failure is silent. The model still generates an answer. It just generates it from incomplete evidence, and you rarely know which questions triggered that path.

Two question types expose this consistently.

The first: questions that require chaining facts across documents. "Which counterparties have exposure to both sector A and sector B, and what is the nature of that exposure?" No single chunk contains that answer. Similarity search retrieves chunks that are semantically close to the question, but none of them span the join. In a credit portfolio context, think of asking which obligors appear across both your ILAAP stress scenario documentation and your individual credit limit approvals. The relationship is real; the text never states it in one place.

The second: synthesis questions about patterns across a whole corpus. "What are the dominant themes in this portfolio of credit memos?" There is no passage to retrieve because the answer does not exist anywhere as text. It emerges only from aggregating across everything.

What a knowledge graph is and what GraphRAG adds

A knowledge graph is a structured representation of entities and the relationships between them. Think of it as a network where nodes are things (companies, people, instruments, concepts) and edges are typed relationships between them: "counterparty A holds instrument B", "regulation C applies to entity D", that sort of structure.

GraphRAG, the approach published by Microsoft Research, uses an LLM to build that graph automatically from your document corpus, then uses the graph to augment retrieval. The key insight is that once relationships are explicit in a graph, you can traverse them at query time rather than relying on similarity search to stumble across them.

It also introduces a second structure on top of the graph: a hierarchy of communities, each with a precomputed summary. That hierarchy is what enables synthesis queries spanning the full corpus, which standard RAG cannot handle at all.

How the index is built

The indexing pipeline is the most important thing to understand about GraphRAG, both technically and economically.

Step one: entity and relationship extraction. For each chunk in your corpus, an LLM reads the text and extracts named entities and the relationships between them. A credit memo might yield: "Borrower: Acme Corp", "Sector: Manufacturing", "Relationship: Acme Corp operates in Manufacturing". This is an LLM call per chunk, over the full corpus.

Step two: graph construction and merging. Extracted entities and relationships are merged into a single graph spanning the entire corpus. Duplicate references to the same entity are resolved, so "Acme", "Acme Corp", and "Acme Corporation" collapse to one node.

Step three: community detection. A graph clustering algorithm partitions the graph into communities: groups of entities that are more densely connected to each other than to the rest of the graph. This produces a hierarchy of communities at different granularities. The original Microsoft implementation used the Leiden algorithm for this step, though the implementation has evolved across versions and the default may vary.

Step four: community summary generation. An LLM writes a summary for each community. This is the second major LLM pass over the corpus, because the summaries are written in advance, not at query time. The output is stored and becomes the basis for global search.

Those four steps mean you are running LLM inference across your full corpus twice before a single user query has been answered.

Two ways to query the graph

GraphRAG exposes two distinct retrieval modes, and choosing the wrong one for your question type is a common mistake.

Local search

Local search is for questions about specific entities and their relationships. When a question arrives, the pipeline identifies the named entities in it, finds those nodes in the graph, and walks their neighbourhood: direct relationships first, then relationships of relationships to the required depth. The retrieved subgraph and associated text chunks become the LLM's context.

This is the mode that handles the chained retrieval failure case. The traversal assembles evidence from multiple sources along typed relationship edges, which similarity search cannot do.

Global search

Global search is for synthesis questions with no specific entity anchor. It runs an operation that maps intermediate answers across the precomputed community summaries and then reduces them into a final response. Each summary gets an intermediate response to the question, and those intermediate responses are then aggregated into a final answer.

This is what makes questions spanning the full corpus tractable. "What are the dominant credit risk themes across this portfolio?" is answered by reading the community summaries, not by searching for similar text. The trade off is latency: you are running LLM inference across potentially many summaries before returning anything.

The cost you are taking on

Indexing cost

The dominant constraint is indexing cost, and it is worth being direct about the scale.

A standard embedding pipeline runs each chunk through an embedding model: fast, cheap, and the same cost regardless of corpus size. GraphRAG runs each chunk through an LLM for extraction and then runs each community through an LLM for summarisation. For a large corpus, the difference can be substantial (think ten times to one hundred times the token spend, depending on corpus size, chunk configuration, and community count). The exact multiple depends on your chunk size, model choice, and community count, but the direction is unambiguous.

Update cadence

Incremental updates are awkward. Adding a document does not just add nodes to the graph: it can shift community structure across the corpus, because community detection is a global operation. In practice this often means full reindexes rather than incremental ones, which compounds the cost further.

Query latency is also higher, particularly for global search. If your application needs responses in under a second, global search is likely a poor fit without significant engineering around caching the intermediate results.

Lighter alternatives worth knowing

Before committing to a full GraphRAG implementation, two patterns are worth considering.

LightRAG is a variant that reduces indexing overhead by simplifying the community layer while retaining entity and relationship extraction. For some use cases it gets you most of the chained retrieval benefit at meaningfully lower cost. It is worth benchmarking against your actual question distribution before assuming you need the full approach.

Graph database with runtime query generation is the simpler pattern: store your data in a structured graph database (Neo4j is the obvious example), and at query time have an LLM translate the user question into a graph query (Cypher, for instance) that runs against the live database. There is no precomputed summary layer and no community detection. It requires cleaner structured data and a good schema, but the indexing cost is far lower and updates are trivial. For finance data with known entity types (counterparties, instruments, positions, regulations) this pattern is often underestimated.

You may also find the post on AI agents in treasury and risk useful context if you are thinking about how retrieval fits into a broader agentic architecture and what that means for your controls.

Is your problem a GraphRAG problem?

Rather than a broad recommendation, here is a short test. Answer these honestly about your actual use case.

Do relationships between entities carry the answer, or just context?

If the answer is "the relationship itself", GraphRAG is a plausible fit. If entities are just metadata around a passage, standard RAG is probably sufficient.

Do your users ask questions that span multiple documents by design?

Not occasionally, but as the dominant query type. If so, local search is worth the indexing cost. If most questions are lookups within a single document or a narrow set, it is not.

Do you need synthesis across the corpus rather than retrieval?

Global search is genuinely useful for aggregated insight questions. If you are building a system where analysts ask summary questions across a large corpus, the precomputed community layer earns its keep. If you are building a document search tool, it does not.

Is your corpus stable, or does it change frequently?

Frequent changes make full reindexing expensive. Be honest about your update cadence before assuming you can afford to run this in production.

Have you tried a graph database with runtime query generation first?

If your domain has known entity types and clean data, that approach is cheaper and easier to maintain. GraphRAG makes more sense when the entity extraction and relationship discovery has to happen from unstructured text and you cannot define the schema in advance.

GraphRAG is a real solution to a real problem. The failure modes of standard RAG on chained and synthesis questions are genuine, not theoretical. But the cost is real too, and for passage retrieval it is simply unnecessary overhead.

If you want to go deeper on building these kinds of pipelines for finance and treasury contexts, the Academy course catalogue covers practical Python and data engineering topics aimed at exactly this audience.