AI Retrieval Architecture
GraphRAG: how AI answers questions across complex data.
A standard retrieval system is good at finding the paragraph that resembles a question. It is less reliable when the answer depends on relationships spread across contracts, reports, tickets, research papers, or database records.
GraphRAG changes the retrieval layer. It extracts entities and relationships, organizes connected information into communities, and gives the language model a structured path through the evidence. The model can answer a narrow question about one entity or summarize patterns across the collection.
That extra structure is powerful, but it is not free. Graph construction adds model calls, storage, update work, and new failure modes. The right architecture uses graph retrieval where relationships matter and keeps simpler search for direct lookups.
Practical Takeaway
GraphRAG is a retrieval decision, not a smarter prompt.
Use GraphRAG when real questions require connected evidence: supplier dependencies, patient histories, legal obligations, research themes, fraud networks, or software impact analysis. Keep vector and keyword retrieval for direct facts. Route each question to the least expensive method that can answer it with traceable evidence.
Retrieval Limit
Similarity search finds nearby text, not the full chain of meaning.
Baseline RAG breaks documents into chunks, converts those chunks into embeddings, and retrieves the nearest matches to a question. That works well when the answer lives in one clear passage.
The weakness appears when the question is relational. A contract names a supplier, a risk report names the supplier's hosting region, and a policy defines which regions require review. None of those chunks may look like the complete question, even though all three are needed for the answer.
Larger context windows do not automatically solve the problem. The system still has to choose what enters the context. Retrieval quality determines whether the model sees the right evidence before generation begins.
A generation failure may really be a retrieval failure.
Before changing the model or prompt, inspect which evidence was retrieved. If the required facts live across several sources and their relationship is missing, the retrieval structure is the real bottleneck.
Indexing Pipeline
GraphRAG turns unstructured documents into a connected evidence layer.
During indexing, the system still keeps source text and chunks. It also extracts entities such as people, products, organizations, locations, systems, and policies. Relationships describe how those entities connect: owns, supplies, depends on, approved by, caused, deployed to, or mentioned in.
The graph is then clustered into communities of closely related entities. Summaries of those communities provide a higher-level view of the collection. This hierarchy lets the query engine work at more than one scale.
Every extracted node and edge should retain provenance back to source text. Without that link, a confident graph-shaped claim can be just as difficult to verify as an unsupported model answer.
{
"entity": {
"id": "supplier:northwind-hosting",
"type": "organization",
"name": "Northwind Hosting"
},
"relationship": {
"source": "product:field-portal",
"target": "supplier:northwind-hosting",
"type": "depends_on",
"evidence": ["contract-18#p4", "architecture-map#node-27"],
"confidence": 0.93
}
}Query Modes
Local search follows entities. Global search summarizes communities.
A specific question usually starts with local search. The system identifies relevant entities, expands to connected neighbors and relationships, retrieves supporting text, and gives that bounded evidence to the model.
A broad question needs a different path. Asking for the major risks across an entire corpus does not contain enough detail for one useful nearest-neighbor search. Global search evaluates community reports, selects relevant findings, and combines them into a whole-dataset answer.
These modes should not be hidden behind one undifferentiated endpoint. Log which route handled the query, what evidence it selected, how much context it consumed, and whether the answer could be verified.
Hybrid Production Design
The practical system routes between retrieval methods.
Production search rarely needs one universal retriever. Exact identifiers belong in keyword or database lookup. Direct semantic questions belong in vector search. Connected and corpus-wide questions are where graph retrieval earns its cost.
A router can begin with deterministic signals: entity identifiers, aggregation language, requested scope, required hops, and data sensitivity. A model classifier can help with ambiguous questions, but its route should be logged and testable.
The final answer layer should receive compact evidence, source identifiers, relationship paths, and instructions to abstain when the graph does not support a claim. Citations need to point to original records, not only to generated community summaries.
Fast path
Use exact and vector retrieval for direct questions with clear evidence.
Graph path
Use local traversal for dependencies, lineage, influence, ownership, and multi-hop questions.
Corpus path
Use community summaries for themes, trends, conflicts, and broad risk review.
Access control must survive graph traversal.
A permitted node can connect to a restricted node. Apply authorization while building query context, preserve source-level permissions, and test for inference leaks across relationships and summaries.
Operating Cost
The graph becomes another production data product.
Graph extraction and community summarization create an expensive indexing stage compared with ordinary chunking and embeddings. Updates are also harder. A changed document can alter entities, relationships, clusters, and summaries downstream.
Entity resolution is a permanent maintenance problem. Product abbreviations, renamed organizations, duplicate customers, and inconsistent identifiers can split one real thing into several nodes or collapse different things into one.
Treat the index as versioned data. Record the extraction model, prompts, schema, source revision, build time, and graph version used for each answer. Rebuild selectively when possible, and keep rollback paths when an extraction change damages retrieval.
Index cost
Entity extraction, relationship extraction, clustering, and summaries add model and compute expense before the first query.
Freshness
Source changes can invalidate edges and community reports, so update strategy belongs in the initial design.
Graph quality
Missing edges, merged identities, and unsupported relationships become retrieval defects that need observability.
Evaluation
Measure whether the graph improves real questions.
Start with a test set from actual user questions. Label which ones are direct lookup, multi-hop, comparison, temporal, or whole-corpus questions. Record the evidence and relationship path a correct answer requires.
Evaluate retrieval separately from generation. A polished answer cannot rescue missing evidence. Track whether the right entities, relationships, source passages, and community reports appeared before scoring answer correctness and completeness.
Compare GraphRAG with the strongest simpler baseline, not a weak demo. Hybrid keyword and vector search with metadata filters and reranking may solve enough of the problem at lower cost. Ship the graph only where it produces a measurable gain.
Architecture Decision
Use GraphRAG when relationships are part of the answer.
GraphRAG is a strong fit for dependency analysis, fraud investigation, biomedical literature, intelligence review, legal discovery, compliance mapping, research synthesis, and enterprise knowledge with meaningful connections across sources.
It is usually unnecessary for a small documentation site, a clean FAQ, or a corpus where nearly every question maps to one passage. Better chunking, metadata, hybrid search, and reranking should be tested first.
The decision is not graph or no graph forever. Build a baseline, classify failures, add the smallest relationship layer that fixes the valuable cases, and preserve a simple retrieval path for everything else.
The shortest useful rule
If users ask 'where is the answer,' improve search. If they ask 'how are these things connected,' evaluate a graph. If they ask 'what does the whole collection say,' evaluate hierarchical summaries.
Related Reading