RetroDeckRETRODECK
Sign inStart free

How to Choose and Size a Vector Database for RAG

Choose a vector database for RAG from your requirements, not a leaderboard: number of chunks, embedding dimensions, latency target, metadata filters, tenancy, and update rate. If the index fits in memory and you already run Postgres, pgvector is usually enough. Move to a dedicated engine when filtered search, hybrid ranking, or scale outgrows it.

Start with six numbers

Write these down before reading any vendor page:

  1. Chunk count, now and in a year. Not documents: chunks, after splitting.
  2. Embedding dimensions, set by your embedding model.
  3. Query rate and latency target, typically a p95 budget for the retrieval step.
  4. Filters: tenant, permissions, date, source, language. How selective are they?
  5. Update rate: append-only, daily re-index, or constant edits and deletes.
  6. Ops budget: can your team run another stateful service, or must this live in something you already operate?

Most RAG projects discover that the corpus is smaller than feared and the filters are harder than expected.

Memory arithmetic

Raw vector storage is chunks times dimensions times bytes per value. Float32 is four bytes, so 10 million chunks at 1,024 dimensions is about 41 GB, and 1 million chunks is about 4 GB. Graph indexes add link storage on top, and the source text and metadata sit alongside.

Three levers shrink it. Half-precision storage halves the bytes. Scalar and product quantization compress further at some loss of recall, which is why engines rescore a candidate set with full-precision vectors. Smaller embedding dimensions, where the model supports them, cut storage linearly. Measure recall after each change on your own queries; the savings are predictable, the quality loss is not.

Index types and what they cost

Flat

Exact brute-force search. Perfect recall, linear cost. Fine for tens of thousands of chunks and useful as the ground truth when you measure an approximate index.

HNSW

A layered proximity graph. High recall and throughput, the default in most products, but memory-hungry and slower to build. pgvector exposes m (default 16), ef_construction (default 64), and hnsw.ef_search (default 40); raising them improves recall at the cost of build time, memory, or latency.

IVF

Vectors are clustered into lists and only the nearest lists are searched. Faster to build and lighter on memory, with recall governed by how many lists you probe. pgvector's documentation suggests rows divided by 1,000 lists up to a million rows, and the square root of rows beyond that, with ivfflat.probes defaulting to 1.

Disk-based and GPU indexes

Milvus documents DiskANN for datasets larger than RAM, noting that disk IOPS can become the bottleneck, plus GPU indexes such as GPU_CAGRA. Its index guide also notes graph indexes usually beat IVF on throughput while IVF variants suit very large top-k and high filter ratios.

Filtering is the real test

RAG almost always filters: by tenant, by permission, by document type, by date. The naive approach searches the index, then drops results that fail the filter. When the filter is selective, that leaves too few results or none. Good engines filter during the search, keep scanning until they have enough matches, or switch strategy for highly selective filters.

pgvector added iterative index scans in 0.8.0 for exactly this: set hnsw.iterative_scan to strict_order or relaxed_order and it keeps scanning when a filter leaves too few rows, bounded by hnsw.max_scan_tuples. Dedicated engines index payload fields so filters can steer the search. Whatever you evaluate, run your most selective real filter, for example one small tenant among thousands, and check both recall and latency.

For strict multi-tenant isolation, a separate collection, partition, or namespace per tenant is often simpler and safer than a shared index with a filter, at the cost of more objects to manage.

Hybrid search

Dense vectors miss exact terms: part numbers, error codes, names. Keyword or sparse retrieval misses paraphrase. Hybrid search runs both and fuses the rankings, and every major product now documents it. What differs is fusion. Qdrant's Query API runs prefetch sub-queries and fuses them with reciprocal rank fusion or distribution-based score fusion. Weaviate fuses BM25F and vector results with an alpha weight, where 1 is pure vector and 0 is pure keyword, and uses relative score fusion by default since v1.24. Pinecone's current docs describe a full-text match filter followed by dense ranking, or client-side fusion with reciprocal rank fusion. Tune the weighting on a labelled query set; defaults are a starting point.

Matching products to situations

Checked against each project's documentation on November 5, 2026.

  • Already on Postgres, index fits in memory: pgvector, at version 0.8.7, with HNSW and IVFFlat. It indexes vector up to 2,000 dimensions and halfvec up to 4,000, and supports sparse vectors. One database, one backup, one permission model.
  • Self-hosted, heavy filtering and hybrid ranking: Qdrant or Weaviate, both built around filtered and hybrid queries.
  • Very large or larger than RAM: Milvus, with the widest documented range of index types, including DiskANN, quantized IVF and HNSW variants, and GPU indexes.
  • No ops capacity: a managed service such as Pinecone, or the managed tiers of the engines above.

The decision is reversible if your pipeline keeps source chunks and metadata outside the vector store and treats embeddings as rebuildable.

What matters more than the database

Retrieval quality is set mostly upstream. How documents are chunked decides whether an answer survives as one retrievable unit. The embedding model decides whether the right chunk is near the query at all. A labelled evaluation set decides whether you know any of this. And a sync process decides whether the index still matches the source next month. Get those right and most vector databases will serve a RAG system well.

Retrieval also sets the cost of the generation step. RDK puts retrieval over private encrypted chunks in front of the model, so repeated or reference-heavy questions are answered from the index and token spend drops 80 to 90 percent.

Frequently asked questions

How big a vector database do I need for RAG?
Multiply chunks by embedding dimensions by four bytes for float32. One million 1,024-dimension chunks is about 4 GB of raw vectors; ten million is about 41 GB. Add index overhead, metadata, and source text, then plan headroom for growth. Half precision or quantization can cut this substantially if recall holds up on your queries.
Should I use HNSW or IVF for RAG?
HNSW is the usual default because it gives high recall and low latency without much tuning, at the cost of memory and slower builds. IVF uses less memory and builds faster, which helps with very large or frequently rebuilt indexes, but recall depends on probing enough lists. Benchmark both on your data against an exact flat search.
Is Postgres with pgvector good enough for RAG?
For many RAG systems, yes. If the index fits in memory and filters are moderate, pgvector keeps vectors next to your relational data with one permission model and one backup process. Its iterative index scans, added in 0.8.0, address filtered queries. Consider a dedicated engine for very large corpora, heavy selective filtering, or demanding hybrid ranking.
Do I need hybrid search for RAG?
If users search for exact terms such as product codes, names, or error messages, yes. Dense embeddings handle paraphrase but blur exact tokens, and keyword retrieval does the reverse. Fusing both, usually with reciprocal rank fusion, tends to improve recall on mixed query sets. Confirm it on labelled queries rather than assuming the gain.