DiskANN hit >5,000 QPS at <3 ms on SIFT1B in NeurIPS 2019; HNSW still leads in RAM-first search for many teams. Read now.
What each algorithm optimizes for
HNSW (Hierarchical Navigable Small World) builds a multi-layer graph in memory. Queries walk from coarse layers down to a dense base layer, following neighbor links that keep average path length short. The design assumes the full graph and vectors fit in RAM, which is why HNSW still leads for many teams that run pure in-memory search and care first about latency under load rather than disk footprint.
DiskANN takes a different constraint seriously: the index and data may not fit in RAM. It stores a graph-structured index on SSD and uses a carefully ordered traversal plus caching so that each query issues a small number of sequential-friendly reads. The NeurIPS 2019 result—more than 5,000 QPS at under 3 ms on SIFT1B—showed that disk-backed ANN can still deliver interactive latency when the access pattern is designed for flash, not for random page thrashing.
What a fair benchmark actually measures
Comparing HNSW and DiskANN without fixing the operating point produces noise. Fix dataset (dimension, metric, cardinality), recall target (for example, recall@10 or recall@100 at a chosen threshold), and concurrency. Then report query latency percentiles, QPS under that recall, build time, peak RAM during build and serve, and on-disk size. A single “winner” number is less useful than a curve of latency versus recall at fixed hardware.
- Hold hardware constant: same CPU, RAM, and storage class (NVMe vs spinning disk changes DiskANN more than HNSW).
- Separate cold vs warm runs: DiskANN benefits heavily from OS and application caches; HNSW is mostly warm after the process is up.
- Report build cost: graph construction and refinement dominate total cost of ownership when indexes rebuild often.
- Match recall before comparing QPS; faster wrong answers are not a win.
Choosing HNSW, DiskANN, or a hybrid
Prefer HNSW when the working set fits comfortably in memory, you need the lowest tail latency under concurrent clients, and you can afford RAM as the primary cost. Prefer DiskANN-style disk-resident graphs when the corpus is far larger than RAM, you still need millisecond-class queries, and you can provision fast SSD with enough IOPS. Many production systems split the problem: hot vectors or recent partitions in HNSW-like RAM indexes, colder bulk in disk-backed structures, with routing that keeps most traffic on the hot path.
Parameter tuning matters more than brand labels. For HNSW, construction degree and search beam width trade build cost and memory for recall. For DiskANN, degree, PQ or other compression, and beam size trade disk reads per query for accuracy. Always retune after a hardware or recall-target change; defaults from one paper’s SIFT1B setup rarely transfer unchanged to your embeddings and QPS goals.
Practical benchmarking workflow
Start with a representative sample of production queries and a labeled ground-truth subset large enough that recall estimates are stable. Build both indexes at several parameter grids, measure recall first, then lock the configurations that meet your minimum recall and re-run for QPS and p50/p95/p99 latency under realistic concurrency. Record peak RSS, disk usage, and cold-start time so capacity planning is honest.
Re-run the suite when embedding models, dimension, or traffic shape change. The 2019 DiskANN SIFT1B numbers prove disk-first ANN can be fast; they do not replace a measurement on your vectors, your filter predicates, and your machines. Treat HNSW vs DiskANN as a capacity and latency design choice, not a permanent ranking—pick the structure that hits your recall SLA at the lowest reliable cost for the data that must stay queryable today.