Building the fastest vector search engine I could

Every time someone opens an item, a recommendation service I work on embeds it and asks for the catalog items nearest to it. A rail of similar items needs ten. A ranking model further down needs 5,000, because it re-scores the whole set before anything is shown.

I built a search engine for that query in Rust. I measured it on 100,000,000 catalog vectors on one machine, against Qdrant, which serves this query today, and against a graph index I wrote on the same kernels.

• Behind HTTP the engine’s lead over the incumbent grew with depth. At K = 10 and recall 0.95 Qdrant was as fast or faster. At K = 5,000 the engine served 10.4 times its throughput, part of it from how I configured Qdrant. Qdrant reached 0.99 at no depth I swept, and the engine at every depth.
• On public data, with the incumbent set up as its documentation says, the lead was smaller. On DEEP-100M, with Qdrant in two segments over gRPC, the engine served 1.9 times Qdrant’s peak at K = 5,000 and recall 0.99. At K = 10 Qdrant served up to 2.1 times the engine’s.
• Against my own graph index the engine won at depth. At K = 5,000 and recall 0.99, in process on the same kernels, it answered 1.9 times the queries of my HNSW, and 3.3 times at 0.995.
• At shallow depth and modest recall the graph is faster. At K = 10 HNSW answered 268,245 queries a second at recall 0.95. The engine’s cheapest setting already recalled 0.989, and at that recall it answered 7% more than HNSW.
• The engine is an inverted file over byte codes with an fp32 rerank. The codes read a quarter of the bytes and the rerank removes the recall ceiling they leave. The index holds 65.4 GB.

A query asks for up to five thousand neighbors

Each catalog item is an embedding of 128 numbers, normalized to unit length, and similarity is the dot product. A query is one more vector of the same kind. The answer is the K catalog items with the highest dot product against it.

The corpus is 100,000,000 items drawn at random from the 104,998,168 live ones. The 4,998,168 left out supply 3,000 item queries, so no query is its own nearest neighbor. A second set of 1,000 text embeddings asks from outside the catalog, and I report it beside the item queries, not mixed in.

Recall is the overlap between the returned set and the exact answer. Order inside the K does not count, because the ranker re-scores the set.

recall@K = | approximate top K  ∩  exact top K | / K

I measured four depths, K = 10, 100, 1,000 and 5,000, and the answer to which index is fastest changes between the first and the last.

How I measured

Everything below ran on the machine in the card, one configuration at a time.

• Every engine got the same cores. Each ran pinned to the same 44 of the machine’s 48. The load generator and the resource sampler had cores of their own, and nothing else ran.
• Every engine started from the same bytes. The corpus sat in RAM on tmpfs. My engines built their indexes from a mapping of it and held them in their own heap, and Qdrant kept its own copy on the same tmpfs.
• Recall carries its spread. Each setting answered all 3,000 item queries in three splits of 1,000. The data section gives the worst split and the 95% interval beside every mean.
• Throughput was measured twice. In process, 44 threads ran queries back to back for 15 seconds after 3 of warm-up. Served, one load generator drove each engine’s HTTP API at 1 to 176 clients.
• Resources came from the kernel. CPU time is from getrusage and /proc, and memory is split into heap, file pages and shared memory. The VM exposes no hardware counters, so there are no cache miss numbers.
• Bandwidth was checked before every in-process configuration. A 44-thread streaming read measured 331 to 346 GB/s each time, so none of those runs started on a degraded machine.

An exact scan reads the whole catalog

The simplest correct engine compares the query with every vector and keeps the best K. It is correct by definition, so it is the ruler for everything else. It also reads all 51.2 GB of fp32 vectors on every query.

One thread took a median 2.23 s per query, measured over 20 queries. With 44 threads the machine finished 17.6 queries a second, 901 GB/s of vectors scored. With every thread reading its own part of memory the machine delivers 360 GB/s, so scans running side by side must share cache lines that a neighbor just fetched.

A read test shows that threads can share reads. Forty-four threads reading the same bytes at the same time read 969 GB/s, and each reading its own part 360 GB/s, averaged over two runs.

I also started each exact scan somewhere else. Each query began at an offset derived from its own values and wrapped around, and recall did not change. At 22 threads the scans finished 6.8 queries a second instead of 9.5, 348 GB/s, about what memory delivers.

Scans started apart seem to drift back into step. At 44 threads they finished 10.3 queries a second, 527 GB/s. In the read test 44 threads started at evenly spaced places read 734 GB/s, more than memory delivers, which suggests they drift together and share lines again.

Byte codes make the exact scan faster and less exact. The same scan over SQ8 codes finished 26.4 queries a second, and its recall stopped at 0.890 for K = 10. Any index that scores these SQ8 codes alone inherits that ceiling.

Three ways to read less

Approximate search buys speed by reading a small part of the catalog, or a small version of it, and accepting an occasional miss. The two designs that dominate skip vectors in opposite ways, and codes shrink whichever vectors remain.

IVF divides the space into cells

An inverted file index (IVF) runs k-means once over the catalog, so every vector belongs to one of nlist cells and sits in that cell’s list. A query compares itself with the cell centers, opens the nprobe nearest cells, and scores every vector in them.

• Its reads are sequential. Each opened list is one contiguous run of memory, read front to back.
• Its scan does not depend on K. Asking for 10 results or 5,000 changes which scored vectors come back, not how many get scored.
• Its misses sit near cell walls. A true neighbor is lost only when it belongs to a cell the query did not open, and opening one more cell usually recovers it.

HNSW walks a graph

A hierarchical navigable small world graph (HNSW) links every vector to a few near neighbors, with sparser layers of longer links on top. A query enters at the top and hops greedily toward its target, dropping a layer each time it stops improving.

On the bottom layer it keeps a working set of the best candidates it has seen. ef_search sets its size.

A graph search can only return what its working set held, so a query for K results needs a working set of at least K. A bigger working set visits many more nodes, and each visit is a read from a random place in memory.

Codes shrink every vector

A code replaces each 512-byte vector with a smaller approximation. A scan over codes reads fewer bytes and computes every score approximately. SQ8 keeps one byte per dimension. Product quantization and RaBitQ keep far less, and need a second pass that rescores a shortlist from a larger copy.

Five settings control these indexes.

Depth decides the index

Each curve in the frontier is one index swept from its cheapest setting to its most thorough, in process on 44 threads, so a query pays for the search and nothing else. Throughput at a given recall is interpolated between the two measured settings either side of it.

The engine in yellow is IVF over 65,536 cells with SQ8 codes and an fp32 rerank. I wrote HNSW on the same fp32 and SQ8 dot-product kernels, so kernel speed does not explain the gap between them.

• At ten results the graph is faster where modest recall is enough. At recall 0.95 HNSW answered 268,245 queries a second. The engine’s cheapest measured setting already recalled 0.989 and answered 43,265, 7% more than HNSW at that recall. HNSW’s sweep ended at 0.989, and at 0.99 the engine answered 40,973.
• At a hundred they meet. HNSW led 3.7 times at recall 0.95 against the engine’s cheapest setting, which recalled 0.977. At 0.99 it answered 33,293 queries a second and the engine 34,126, and above 0.992 only the engine reached.
• At a thousand the engine leads from 0.99 up. It answered 24,613 queries a second at 0.99 against 20,115, and 18,665 against 5,531 at 0.995.
• At five thousand the engine leads at every recall. A graph cannot carry a working set smaller than K, so HNSW’s cheapest setting here was ef_search = 5,000. The engine answered 11,428 queries a second at 0.99, 1.9 times HNSW, and 3.3 times at 0.995.

If 0.95 is enough at K = 5,000, the fp32 copy is not needed at all. SQ8 codes alone on the same 65,536 cells answered 27,734 queries a second at that recall from 14.2 GB, three times the fastest HNSW.

Text queries are harder for both indexes. At K = 10 the engine at 8 cells recalled 0.869 of the true ten for text queries against 0.989 for item queries, and HNSW at ef_search = 640 recalled 0.884 against 0.989.

The exact SQ8 scan went the other way on text queries, 0.954 against 0.890, so the loss comes from which vectors the indexes skip and not from the code.

Behind HTTP the gap to Qdrant grows with depth

In process a query pays only for the search. A service also parses the request, waits for a thread, and serializes K results back. I ran the engine and Qdrant behind their HTTP APIs on the same 44 cores, and drove both from one load generator on two other cores.

• The incumbent, with its default graph. Qdrant 1.19.0, one collection, dot product, M = 16 and ef_construct = 100, in three segments. Its fp32 vectors were mapped from tmpfs and an int8 copy was held in RAM. I measured it searching fp32, and searching int8 with a rescore at twice oversampling.
• The incumbent, rebuilt with my graph’s settings. A second Qdrant collection with M = 16 and ef_construct = 200, the settings my HNSW used, searched in fp32 with ef_search up to 2,048 at K = 10 and 40,000 at K = 5,000.
• The engine as proposed. 65,536 anisotropic cells, SQ8 codes and a rerank of max(2K, 200). It answered in JSON as Qdrant does, and once more in a binary format.
• The same response per request. Both returned ids and scores and no payloads. The load generator built every request body before the clock started and parsed recall from every response.

• The denser graph lifted Qdrant’s recall. At ef_search = 512 it raised recall at K = 10 from 0.954 to 0.977, and it reached 0.983 at 2,048, still rising. At K = 5,000 it flattened at 0.982. My HNSW with the same graph settings reached 0.989 at K = 10 and 0.996 at K = 5,000.
• At ten results Qdrant was as fast or faster. With the denser graph at recall 0.953 it served 40,745 queries a second in the sweep and 50,845 in a second pass, against the engine’s 41,196 at 0.989. At recall 0.90 it served 49,127, more than the engine at any setting I served.
• The lead grows with depth. Against the faster Qdrant at recall 0.95 the engine served 1.6 times as many queries at K = 100, 5.1 times at K = 1,000 and 10.4 times at K = 5,000.
• The lead grows with recall. At 0.98, which only the denser graph reached, the engine served 4.4 times Qdrant’s throughput at K = 10 and 12.1 times at K = 5,000.
• Qdrant’s throughput with many clients varied. With its default graph at K = 100 it answered 10,217 queries a second with 44 clients and 4,140 with 176. The denser graph slowed past 44 clients at K = 10 in the sweep and not in the second pass.
• At depth Qdrant spent ten times the CPU per query. At K = 5,000 with 44 clients it used 36.9 ms a query against the engine’s 3.7 ms, at recall 0.965 and 0.971, and its p99 was 46.8 ms against 7.6 ms. Its JSON response was 250,622 bytes against 96,226.
• Part of that gap is how I configured Qdrant. I capped its segments at 40,000,000 points so a rebuild fit on tmpfs, so it searched three segments, each with ef_search of at least K. My HNSW with the same graph settings used 6.5 ms a query at ef_search = 5,000. I did not test one segment or Qdrant’s gRPC API.
• The HTTP path cost the engine little. At K = 10 it served 41,196 queries a second against 43,265 in process. Binary responses served 7% more than JSON at K = 5,000.
• Memory differed by what each holds. Qdrant’s heap was 21.9 GB and its mapped storage 57.6 GB. The engine’s heap was 65.8 GB, and it also mapped the 51.2 GB corpus it built from, which a replica loading a snapshot does not need.

Loading 100,000,000 points into Qdrant over its HTTP API took 25 minutes, from a Python client on four cores.

Indexing them one segment at a time, the workaround under what I got wrong, took about 18 more with the default graph and 26.5 with the denser one. These times measure my setup more than Qdrant. The engine built from the mapped corpus in 12.7 minutes.

Byte codes need a second pass

Every index that scores codes alone stops short of full recall, and where it stops depends on K. The exact SQ8 scan has the same ceiling, so the index structure is not the cause.

The true neighbors are closer together than the code can resolve

SQ8 stores each dimension as one of 256 levels between that dimension’s calibrated minimum and maximum. I measured what that does to a score on the true top 100 of all 3,000 item queries. The median score moved by 0.00050, and one in ten moved by more than 0.0012.

The catalog is dense near a typical item query. The ten true neighbors of a median query span 0.0091 of score, and the gap between the tenth and the eleventh is 0.00030. A code error larger than the gap can swap neighbors across the boundary, and the exact SQ8 scan returns 0.890 of the true ten.

The gap shrinks with K and the ceiling still rises. A code error can only swap a neighbor that sits within about one error of the K-th score. For a median query that band holds 20% of the true ten and 1.8% of the true 5,000.

A shortlist rescored in fp32 removes the ceiling

The fix is a second pass. Scan the codes, keep a shortlist of rerank_n = max(2K, 200) candidates, rescore those from full vectors, and return the best K.

On the same 16,384 cells with 8 opened, the rerank lifted recall at K = 10 from 0.887 to 0.994 and cost 3% of the throughput. IVF over fp32 needs the same 8 cells for 0.995, and the codes answered 2.9 times as many queries because they read a quarter of the bytes.

Deeper requests pay more for the second pass. At K = 5,000 the shortlist is 10,000 vectors, 5.1 MB of random reads per query.

With 32 cells opened the rerank lifted recall from 0.982 to 0.996 and took 27% of the throughput. It still answered 2.3 times the queries of IVF over fp32 on the same 32 cells, which reached 0.997.

Smaller codes lose at depth

I built the same 16,384 plain k-means cells with five codes and scanned each the same way. The two codes smaller than SQ8 rescore a shortlist from a larger copy. Their shortlists held at least 200 candidates and up to eight times K.

• LVQ8 gives every vector its own range. That raised the ceiling to 0.923 at K = 10 and cost a quarter of the throughput. It is the one code here that beat SQ8 on recall.
• TQ8 rotates the space before it quantizes. On this catalog it lost recall everywhere, and its build took 915 s against SQ8’s 187.
• PQ and RaBitQ need shortlists too long to rescore at depth. At K = 5,000 even 40,000 rescored candidates left RaBitQ at 0.947. At K = 10, with 200 candidates rescored, they returned little more than half the true ten.

Better cells scan fewer vectors

• Anisotropic k-means evens the lists out. Plain k-means left its largest list at 160,573 vectors, 26 times the average. Trained with the anisotropic loss the largest was 35,422. At K = 5,000 with 16 cells opened, the even lists scanned 20% fewer vectors for the same 0.974 recall.
• More cells helped less than the codes did. With the rerank at recall 0.99, 65,536 cells answered 1.12 times the queries of 16,384 at K = 5,000 and 1.25 times at K = 1,000, and matched them at K = 100. They took 3.7 times as long to build.

How the engine works

The engine is Rust with no dependencies outside the standard library, and one binary is both the index builder and the HTTP server.

The build lays each cell out as one run of memory

Building the index is four passes, each spread over the 44 cores.

1. Train the cells. k-means on a random sample of 1,000,000 vectors finds 65,536 centroids. The loss is the anisotropic one from ScaNN, which counts error along a vector four times as heavily as error across it.
2. Assign every vector. Each of the 100,000,000 goes to the centroid with the highest dot product. This pass is most of the build, 658 of its 762 seconds.
3. Calibrate the code. SQ8 takes each dimension’s range from a strided sample of 50,000 vectors. 0.0041% of all codes fall outside it and clip.
4. Write the lists. The builder writes vectors in cell order into three aligned arrays, SQ8 codes at 128 bytes, fp32 vectors at 512 and ids at 8. A cell is one contiguous run in each.

The builder stores the centroids as int8 as well. The table is 8.4 MB, small enough to stay in the 80 MiB L3 cache.

A query is one pass over a few cells

One thread answers one query, and a query makes four steps.

1. Route. The worker quantizes the query to int8, scores it against all 65,536 centroids with the Arm sdot instruction, 16 multiply-adds at a time, and keeps the best nprobe cells.
2. Scan. The worker folds each dimension’s SQ8 scale into the query’s weights once, so a code costs one widen and one fused multiply-add. The kernel scores four vectors per pass and reuses every weight it loads across all four.
3. Shortlist. Each score is packed with the vector’s position into one 64-bit key. A buffer keeps the best rerank_n, is cut back with a partial sort when it fills, and rejects any later key below the cut with one comparison.
4. Rerank. The shortlist is rescored from the fp32 array and the best K return. These are the only random reads a query makes, 512 bytes per candidate.

The HTTP server is part of the same binary. Connection threads read requests into a bounded queue, and a fixed pool of worker threads runs the searches. When the queue is full the server answers 503 at once instead of letting latency pile up behind it.

What it costs to hold, query and build

• Memory is mostly the fp32 copy. The engine holds 65.4 GB, and 51.2 GB of that is the fp32 array the rerank reads. HNSW over fp32 holds 60.5 GB, 8.5 GB of it graph links. IVF over SQ8 holds 14.2 GB in all.
• The engine spends less CPU per query. At the first measured setting above recall 0.99 for K = 5,000, a query cost the engine 3.9 ms of CPU and HNSW 9.4 ms. Both kept all 44 cores busy, so CPU per query is what set their throughput.
• The build is mostly assignment. The engine built in 12.7 minutes on 44 cores. HNSW took 20.4, because it inserts each vector with a search of its own.

The pieces come from FAISS and ScaNN

An inverted file over scalar-quantized codes with an exact rerank of a shortlist is the design of FAISS’s IndexIVFScalarQuantizer wrapped in IndexRefineFlat, described in the Faiss library paper. The loss that trains the cells is the anisotropic quantization loss from ScaNN.

• What I changed is tuning for depth. The shortlist keeps max(2K, 200) candidates, a floor that matters at shallow K. The centroids are scored as int8, 16 multiply-adds an instruction, and the shortlist is a buffer of packed 64-bit keys cut back with a partial sort.
• Graph indexes take the other road. HNSW and DiskANN walk a neighbor graph, and SPANN keeps inverted lists on disk with balanced clustering. The graph results in this post come from my own HNSW.
• Published comparisons stop at shallow depth. The big-ann-benchmarks competitions scored recall at K = 10. Depth changes which design is fastest, which is the result this post is about.

The proposed service is a fleet of identical replicas

At 100,000,000 vectors the whole index fits in one machine’s memory, so the service needs no sharding. Every replica holds a full copy, any replica can answer any query, and capacity should grow by adding copies. Of this design I measured the build, one replica’s capacity, and the load of a smaller SQ8 snapshot.

A query reads one replica

The balancer keeps no state, because queries are independent and replicas are identical, so it can send each query to any healthy replica. There is no scatter and no merge at this size. The balancer is designed and not built.

Inside a replica, connection threads read requests into a queue of 4,096 slots, and 88 worker threads take them one query each. When the queue is full the server answers 503 at once, so a caller can retry on another replica instead of waiting. No test filled the queue, so that path is untested.

A worker then routes, scans, shortlists and reranks as the previous section describes. At K = 5,000 with 48 cells opened, a query scanned 82,473 codes, 10.6 MB, reranked 10,000 fp32 vectors, 5.1 MB, and cost 4.4 ms of CPU behind HTTP, 3.9 ms in process. The response was 96,226 bytes of JSON.

A build produces a versioned snapshot

• The builder is one job. It reads an export of the catalog’s embeddings, trains the cells, assigns every vector and writes the lists, 12.7 minutes on 44 cores.
• A validator gates the release. It runs held-out queries against their exact truth, computed from the same export, and releases the snapshot only if recall matches the last one. This step is designed and not built.
• The store keeps every version. The index is 65.4 GB in memory, and a snapshot would be about that size. The store is designed and not built.

A rollout swaps snapshots in place

A replica’s index takes 65.4 GB of the machine’s 188 GB, so a replica can load the new version beside the old one and swap them without leaving the balancer. Replicas swap one at a time.

The loader reads every section into memory and verifies a checksum on each. A 14.0 GB snapshot of an SQ8 index on 16,384 cells had answered by the second health check, 2.0 s after the process started, so it loaded from tmpfs at 6.9 GB/s or faster. Fetching from the store is not included.

At that rate the full 65.4 GB would take under 10 s, which I estimated and did not measure.

The registry that tells replicas which version to load, the in-place swap and the rollback are designed and not built. Rolling back is pointing the registry at the previous version.

New items go to a delta

A new item goes to the cell whose centroid is nearest, into a small delta list beside that cell’s main list. A query scans the delta lists of the cells it opens, so the item is searchable at once. A delete is a tombstone the shortlist checks, and the next build folds both into the lists.

The delta costs little. With a million new items spread like the catalog, a query scans about 820 more vectors, 1% of the 82,473 it already scans, an estimate I did not measure. The change log and the delta are designed and not built.

A failed replica drops out on its own

The balancer checks every replica’s health and stops sending queries to one that does not answer. A replacement loads the current snapshot from the store and rejoins, the same path as a rollout. Of this, the bounded queue and its 503 are built, though no test filled the queue. The health checks and the replacement are designed.

How many replicas a load needs

A replica’s capacity is the highest arrival rate it keeps up with, not the peak a closed loop reaches. I drove both servers open loop, with requests arriving on a random schedule over up to 512 connections, whether or not earlier ones had returned.

Each request was timed from its scheduled arrival. A server that falls behind shows it as requests the load generator could not send, because every connection was waiting.

An engine replica keeps up until its cores are full, then falls behind within one step. At K = 5,000 and recall 0.99 one engine replica kept up with 10,054 queries a second, with all 44 cores busy and p99 at 52.7 ms. At 1.1 times that rate it left 22,230 requests unstarted in 20 seconds.

I size each fleet at 70% of the rate one replica kept up with, the same share for both servers. At that rate the engine’s p99 was 5.5 ms at K = 5,000.

• At depth one engine replica does the work of more than ten Qdrant replicas. At K = 5,000 the engine kept up with 12,242 queries a second at recall 0.971 and Qdrant with 1,161 at 0.979, since Qdrant cannot search below ef_search = K. For 20,000 a second that takes 3 engine replicas or 25 Qdrant replicas.
• At ten results the fleets are close. One engine replica or two Qdrant replicas carry 20,000 queries a second at recall 0.95. At recall 0.99 the engine needs one replica at K = 10 and three at K = 5,000, and Qdrant reached 0.99 at no depth.
• Qdrant’s shallow limits look like stalls, not saturation. At K = 10 it kept up at half its peak and fell behind at 70%, with 19 of 44 cores busy, a median of 1.2 ms and a p95 of 11.7 s. At K = 1,000 it fell behind at its peak rate and kept up again at 1.1 times it.
• I did not diagnose the stalls. With 176 clients the K = 10 setting answered 6,782 queries a second in one run and 50,845 in the next. I ran Qdrant’s default service settings and did not tune its threads, so its fleets at K = 10 to 1,000 rest on stalls I cannot explain.
• At depth Qdrant’s limit was its cores. At K = 5,000 it fell behind with every core busy.
• The price is a list price. Each replica is one c4a-standard-48 at $2.1552 an hour on demand in us-central1, from cloudprice.net. A discount lowers both fleets alike and leaves the ratio where it is.

Each design decision had a test

I wrote each design decision down as a hypothesis and tested it. Two of them came after a result surprised me, the cache sharing in the exact scan and Qdrant’s graph.

How I checked the numbers

• The truth was checked twice. An exact fp32 scan produced the true neighbors, and an independent float64 scan agreed on all 5,000 neighbors of five queries. A second round of runs a day later recomputed the truth for 100 queries, and it matched the first byte for byte.
• Served recall matched in-process recall at the setting I compared. At K = 10 with 8 cells opened, recall parsed from HTTP responses was 0.9887, the same as in process.
• A rerun a day later matched. Behind HTTP the engine peaked at 41,177 queries a second at K = 10 with 8 cells, against 41,196 the day before.
• Repeated settings agree. At K = 10 several shortlist sizes round up to the same 200 candidates, so 34 settings ran two or three times each. Throughput differed by at most 0.5%, with a median of 0.04%, and recall was identical every time.
• Ties do not move the answer. Counting any result within 1e-6 of the K-th true score changed no recall by more than 0.002.
• The prose is recomputed. The site’s tests read the committed result files and fail if a pinned number drifts from them.

What I got wrong

• The first exact-scan run undercounted. Its load window dropped queries that straddled either edge, and at 2.2 s a query that hid 16% of the throughput, 14.7 against 17.6 queries a second. I fixed the window in both load generators and ran it again.
• Qdrant’s first index build failed. It needs about twice a segment’s size free to rebuild it, and the tmpfs could not hold that beside the collection. The data section describes how I rebuilt one segment at a time.
• An earlier version of this post rested on a smaller study. It timed one engine in process and Qdrant through a client, on a sample of the catalog. Every number here is new.

A public dataset shows the same crossover

Every number above is on a catalog nobody else can download. So I ran the in-process comparison again on DEEP-100M, the first 100,000,000 vectors of the public DEEP-1B set, with its 10,000 queries. My exact ground truth matched the official one on 99.995% of the top 100 neighbors.

• The order still reverses with depth. At recall 0.95 my HNSW answered 2.0 times the engine’s queries at K = 10, and the engine answered 2.1 times HNSW’s at K = 5,000. At K = 1,000, where the two were within 5% on the catalog, the engine led by 1.3 times.
• The lead at the deep end matched. At K = 5,000 and recall 0.99 the engine answered 1.9 times HNSW’s queries on DEEP, the same ratio as on the catalog.
• Each query reads far more of this dataset. On the catalog, recall 0.99 at K = 5,000 took 48 cells, about 82,000 vectors. On DEEP, 512 cells and about a million vectors reached 0.970, and 0.99 took between 1,024 and 1,536 cells. There the engine answered 1,366 queries a second, an eighth of its catalog rate.
• The same design runs slower in the library. FAISS 1.10.0 searched IVF65536,SQ8 with a flat refine at the engine’s cell count and rerank depth. It answered 2.4 to 2.6 times fewer queries at every depth and recall both reached.
• The anisotropic cells did not help here. FAISS trained plain k-means on four times the sample, and its cells found slightly more neighbors each, up to 0.017 of recall at 8 cells.
• Behind their APIs the lead shrank. I served Qdrant in two segments over gRPC. At K = 5,000 and recall 0.95 the engine served 4.4 times its peak, against 10.4 on the catalog, and at 0.99, which Qdrant never reached on the catalog, 1.9 times. At K = 10 Qdrant served up to 2.1 times the engine’s.

Tuning for public data made the engine a hybrid

At full precision, with no byte codes, the engine needed more machines than the incumbent in 14 of the 16 depths and recalls I sized across both public datasets. Its scan already read memory at 340 to 365 GB/s, the machine’s limit, so it could only get faster by reading less.

I changed one thing at a time, at full precision and in process, on Cohere 10M and on DEEP-10M, the first 10,000,000 vectors of DEEP-100M.

• More training changed nothing. Twenty k-means iterations instead of four moved throughput by under 4% at every depth and recall on both datasets.
• Filing by dot product lopsides the cells. Plain k-means that filed each vector by its highest dot product left one Cohere cell with 442,026 vectors, where the median cell held 196. Filing and routing by distance kept the largest at 4,087 and ran 11% faster at K = 10 and recall 0.95.
• More cells helped the cells most. Four times as many, 65,536 for 10,000,000 vectors, doubled throughput at K = 10 and recall 0.95 on Cohere and raised it 1.3 times on DEEP-10M.
• Spilling helped in one place. Storing each vector in a second cell ran 18% faster on Cohere at K = 10, and about half as fast on DEEP-10M.
• A graph with more links beat every cell layout on Cohere. At recall 0.99 a graph with 48 links a node answered 9,481 queries a second at K = 10, 16 times the best cells, and 3.8 times them at K = 5,000. With 16 links, built on 44 threads, it topped out at 0.985.
• On DEEP-10M the cells still won at depth. At K = 5,000 and recall 0.95 the best cells answered 5,595 queries a second to the best graph’s 2,557, and at 0.99 the two were within 7%.

So the engine’s full-precision default is now a hybrid: a graph with 32 links a node for all but the deepest queries, and 65,536 cells beside it for those. 32 links kept most of 48’s gain and built in two thirds of the time.

What this does not show

• Two datasets, and no text embeddings. The catalog and DEEP-100M are both item or image vectors. The code ceilings and the cells a query must open come from how densely neighbors pack around a query, and a text embedding model packs them differently.
• No filters and no writes. Every query searched the whole catalog, and nothing changed while it served. Qdrant does both today. The engine does neither yet.
• One machine, on loopback. There were no replicas, no network hop and no failover. The fleet sizes multiply one replica’s measured rate, and the load generator shared the machine on cores of its own.
• Few graph configurations. My HNSW ran at M = 16 and ef_construct = 200. I measured Qdrant at its defaults and at ef_construct = 200, both with M = 16. A larger M may raise both ceilings, at a cost in memory and build time.
• Qdrant as I configured it. Three segments set by my tmpfs limit, the REST API and not gRPC, default threads, and open-loop stalls I did not diagnose. One segment may change both its CPU per query and its recall.
• No saved snapshot of the proposed index, and no full queue. I loaded a smaller SQ8 snapshot, and no run filled the engine’s queue.
• One load generator. Every open-loop rate came from one generator on two cores holding 512 connections. A fleet behind a real balancer adds that balancer’s own queueing, which none of this measures.
• No hardware counters. The VM exposes none, so the cache sharing in the exact scan rests on bytes, time and the offset test, not on measured misses.