Two-stage retrieval can cut multi-hop miss rates while holding p95 latency under 300 ms in tuned RAG pipelines. Read the full breakdown now.
Why pure vector search fails multi-hop questions
Most RAG stacks embed a query, retrieve nearest neighbors, and pass those chunks to a model. That works when the answer lives in one passage. Multi-hop questions break the pattern: the first fact is needed to find the second, and dense similarity alone rarely bridges that gap. A query about “which systems depend on the service that owns this failing queue” may rank well on the queue name and still miss the ownership and dependency passages that actually resolve the question.
Graph retrieval helps because relationships are explicit. Nodes and edges encode ownership, dependency, version lineage, and document structure. Alone, graphs struggle with fuzzy language and paraphrase. Alone, vectors struggle with structured traversal. Hybrid search uses each where it is strong: vectors to enter the right neighborhood, graphs to walk the paths that multi-hop answers require.
Two-stage retrieval that stays under a 300 ms p95 budget
A practical hybrid design is two stages. Stage one is a fast vector (or sparse-plus-dense) recall over chunks and, when useful, over node text. It returns a short candidate set—enough to cover likely entry points without flooding the graph. Stage two starts from those candidates and expands along typed edges: document sections, entity links, citation edges, or domain relationships. Expansion is bounded by hop count, edge type filters, and a hard node budget so the second stage cannot dominate latency.
In tuned RAG pipelines, this pattern can cut multi-hop miss rates while holding p95 latency under 300 ms. The budget holds when stage one is cached or ANN-indexed, stage two runs against a local or co-located graph store, and both stages share a single request path with strict timeouts. Measure end-to-end p95 including graph expansion and reranking, not only the first vector call.
What to store, score, and return
Store chunks as vector documents and mirror entities or sections as graph nodes with stable IDs. Edges should carry type and weight; “mentions,” “depends_on,” and “section_of” behave differently at expand time. When stage two returns nodes, resolve them back to chunk text (or summaries) so the generator still sees readable context. Prefer returning a small, ordered context pack over dumping every neighbor.
- Score stage-one hits with vector or hybrid lexical-dense scores.
- Boost stage-two neighbors by edge type relevance and hop distance decay.
- Rerank the merged set once, then cap tokens for the prompt.
Deduplicate aggressively: the same paragraph may appear as a vector hit and again as a graph neighbor. Without dedupe, you waste context window and inflate latency for no gain.
Operational checks before you ship
Build an evaluation set that separates single-hop and multi-hop questions. Track miss rate per class, not only aggregate recall. Log stage latencies separately so you know whether p95 pressure comes from ANN, graph hops, or rerank. Gate expansion: if stage one confidence is high and the query is clearly single-hop, skip the graph path. If stage one is empty or low-scoring, widen recall before expanding.
Start narrow—one or two edge types that match your domain—then widen. Hybrid search pays off when multi-hop misses matter more than raw simplicity. Keep the pipeline boring: bounded hops, shared IDs, one rerank, and a hard latency budget you actually measure in production-like traffic.