Scaling Vector Search in Production

Every RAG (retrieval-augmented generation) application starts the same way — a small vector database, a few thousand embeddings, everything fast because the dataset is tiny. The interesting engineering decisions show up later, once you're past a few million vectors and the choice of vector database starts actually mattering for latency and cost. Here's how the major options compare in production.

The three realistic starting points

pgvector — a Postgres extension — is the strongest default for most teams already running Postgres, specifically because embeddings, source documents, and metadata all live in one database you can query with ordinary SQL joins. No separate system to operate, no data synchronization problem between your primary database and a separate vector store.

Pinecone is fully managed with no infrastructure to operate — you get low latency (roughly 8ms p50) without running anything yourself, which matters for teams that don't want to own vector database operations at all.

Qdrant is a purpose-built vector database with the lowest latency among the options generally benchmarked (around 4ms p50, 25ms p99), and it can be self-hosted or run as a managed service.

Where the performance gap actually shows up

At smaller scale — under roughly 10 million vectors — all of these perform adequately, and the choice matters less than picking whichever fits your existing infrastructure. The gap widens as you scale: at around 5 million vectors, pgvector's p95 latency climbs into the 80-140ms range depending on search parameters, while Pinecone holds under 30ms on pod-based deployments. Past 10 million vectors and up toward a billion, the realistic field narrows mainly to managed Pinecone or self-hosted purpose-built options like Qdrant, Weaviate, and Milvus.

Metadata filtering is a bigger differentiator than raw latency

A detail that matters more in practice than the headline latency numbers: how each database handles filtering search results by metadata (e.g., "find similar documents, but only from this user's account"). Qdrant's filtering happens inside the HNSW graph traversal itself, which is materially faster for selective filters than the alternative. pgvector and some purpose-built databases apply metadata filters as a post-processing step on the initial candidate set — meaning if your filter is highly selective (only a small fraction of vectors match), you can end up scanning far more candidates than necessary before filtering down, which shows up as real latency at scale.

If your application relies heavily on metadata-filtered search — most multi-tenant SaaS RAG applications do, since you almost always need to scope results to a specific user or account — this operational detail matters more than the raw unfiltered latency benchmark most comparisons lead with.

Cost reality at meaningful scale

For a representative enterprise workload (around 10 million vectors at 1536 dimensions, a common embedding size), self-hosted Qdrant lands in a roughly $400-950/month range depending on configuration, with managed Qdrant options running somewhat less. Pinecone's serverless pricing scales with read-unit consumption tied to index size, which can be cost-effective at lower query volume but grows less predictably as both data and query volume scale together.

The practical decision framework

• Already on Postgres, workload under a few million vectors, modest query volume — pgvector is the right default. The operational simplicity of one database outweighs the latency gap at this scale.
• Need zero operational overhead and can accept Pinecone's pricing model — a reasonable choice for teams that want to never think about vector database infrastructure.
• Scaling past 10 million vectors, or metadata filtering performance matters a lot — Qdrant (or a comparable purpose-built option) is worth the added operational complexity of running a dedicated vector database.

Quantization is the other lever besides picking a database

Beyond choosing which database to run, one of the most impactful production techniques for controlling both memory footprint and cost at scale is vector quantization — compressing the stored representation of each vector rather than switching databases entirely. Scalar quantization (mapping each dimension down to int8 or fp16) typically achieves around 75% memory reduction with minimal recall loss. Binary quantization goes much further, compressing each dimension to a single bit — a 768-dimension vector shrinks from 3,072 bytes down to 96 bytes, a 32x compression ratio. At real scale, that difference is dramatic: a 100-million-vector index at 768 dimensions can shrink from roughly 367GB down to around 38GB with binary quantization, small enough to fit comfortably in the buffer cache of a standard database instance rather than requiring expensive high-memory hardware.

The catch is recall — compressing that aggressively loses precision, so the standard production pattern is to search the compressed, in-memory index first to get a candidate set, then re-rank that smaller candidate set against the full-precision vectors (typically oversampling by 3-4x) to recover 95-99% of the recall you'd get from an uncompressed search. Current guidance splits roughly by scale: for workloads in the 5-50 million vector range, int8/scalar quantization with reranking is close to a "free lunch" — a 4x memory reduction with negligible recall impact. Past 50 million vectors, or in latency-bound scenarios where memory cost dominates the budget, binary quantization with oversampling and reranking becomes the more common choice, trading a bit more engineering complexity for an order-of-magnitude memory reduction. This is worth evaluating before assuming you need to migrate to a different database entirely — quantization can meaningfully extend how far pgvector or any of the other options here scale before the underlying database choice itself becomes the bottleneck.

The practical takeaway

Don't pick a vector database based on benchmark headlines alone — the operational cost of running a separate system, and how your specific workload interacts with metadata filtering, matter more in practice than raw p50 latency numbers. Starting with pgvector if you're already on Postgres is a reasonable default for most RAG applications; migrate to a purpose-built option only once you have concrete evidence — real latency numbers from your own workload — that you've outgrown it.

Sources: kalviumlabs.ai, digitalapplied.com, knowsync.ai