Choosing a Vector Database

Why this question comes up so often

Anyone building a retrieval system or a retrieval-augmented generation (RAG) pipeline eventually hits the question: which vector database should I use? There are now many options, each with its own marketing page claiming to be the fastest or the most scalable. The honest answer is that the choice depends on a small number of concrete factors about your data and your workload, and most of those factors can be reasoned about before you touch any product. This article builds up the ideas from the ground, so that by the end you can look at your own project and make a grounded decision rather than a guess based on which name you saw most recently.

Start with the problem, not the product

Traditional search matches text against text. If you search a document store for the word invoice, you get back documents containing that word, perhaps with some stemming or synonym handling. This breaks down when the meaning you want is not expressed in matching words. A customer support query like my package never arrived should also match a document titled shipment delays and refund policy, even though the two phrases share almost no words.

Embeddings solve this by mapping text (or images, audio, or other data) to a vector of numbers, produced by a trained model, such that items with similar meaning end up close together in that numeric space. Two sentences about a lost package will have vectors that point in a similar direction, even if the wording differs completely. Once you can represent meaning as a point in space, search becomes a geometry problem: given a query vector, find the stored vectors that are closest to it.

A small worked example

To make this concrete, imagine a tiny toy embedding space with only two dimensions (real embedding models typically produce hundreds or thousands of dimensions, but two dimensions let us see the idea clearly). Suppose five words have been embedded as vectors, and we want to find which ones are closest in meaning to a query vector that sits near the direction associated with cat.

import numpy as np

# Toy 2D embeddings (in practice these would come from an embedding model
# and have hundreds of dimensions)
vectors = np.array([
    [0.80, 0.60],   # cat
    [0.90, 0.50],   # kitten
    [-0.50, 0.80],  # car
    [0.10, -0.90],  # ocean
    [0.85, 0.55],   # puppy
])
labels = ["cat", "kitten", "car", "ocean", "puppy"]

query = np.array([0.82, 0.58])  # a query vector close to "feline"

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

scores = [cosine_similarity(query, v) for v in vectors]
ranked = sorted(zip(labels, scores), key=lambda x: x[1], reverse=True)

for label, score in ranked:
    print(f"{label}: {score:.3f}")

Running this (tested with numpy 1.26) prints the labels ranked by cosine similarity to the query. You should see cat and puppy at the top with scores very close to 1.0, kitten just behind them, car much lower at around 0.06, and ocean negative at around minus 0.48, meaning it points in roughly the opposite direction. This is the entire idea behind vector search: turn items into vectors, turn the query into a vector using the same model, and rank stored items by a similarity score such as cosine similarity.

Notice that this toy example used plain numpy, comparing the query against every stored vector directly. This is called brute-force or exact nearest neighbour search, and it is worth remembering because it is a perfectly valid approach when the number of vectors is small, say in the tens of thousands. A vector database only starts to earn its keep when exact comparison against every vector becomes too slow or too memory-hungry to do on every query.

What a vector database actually is

A vector database is a system designed to store large numbers of vectors together with associated metadata, and to answer nearest-neighbour queries quickly, without comparing the query to every single stored vector. It typically provides four things that a plain array or a homemade numpy script does not provide on its own: an approximate search index that trades a small amount of accuracy for a large amount of speed, persistence so your data survives a restart, the ability to filter results by metadata (for example, only search within documents owned by a particular user), and operational features such as replication, backups, and a network API so multiple applications can query it concurrently.

It helps to be precise about the word approximate here. Once you have millions of vectors, comparing a query against every one of them (exact search) becomes too slow for an interactive application. Vector databases instead build an index structure that lets them skip most of the vectors while still finding results that are very likely, but not mathematically guaranteed, to be the true closest matches. This trade-off between speed and perfect accuracy is central to how these systems work, and it is the reason the field talks about recall rather than correctness: recall measures what fraction of the true nearest neighbours the approximate search actually returns.

Core concepts you need before comparing products

Distance metrics

Vectors are compared using a distance or similarity metric. The three you will meet most often are cosine similarity (the angle between two vectors, ignoring their length, as used in the worked example above), dot product (similar to cosine but also sensitive to vector magnitude), and Euclidean distance (straight-line distance between the two points). Which one is correct depends on how the embedding model was trained. Many popular text embedding models are trained and evaluated using cosine similarity, so using Euclidean distance against them can give subtly wrong rankings. This is a detail that is easy to overlook and worth checking in the documentation of whichever embedding model you use.

Approximate nearest neighbour indexes

Two families of indexing approach show up repeatedly across vector databases, and understanding the intuition behind them helps you read any product's documentation without feeling lost.

The first family is graph-based, and the most widely used example is HNSW (hierarchical navigable small world). Picture every stored vector as a node, connected to a handful of its nearest neighbours by edges, with a few shortcut edges connecting distant parts of the space. Searching means starting at an entry point and repeatedly hopping to whichever connected neighbour is closer to the query, somewhat like using connecting flights to get close to a destination before switching to local roads. This tends to give very good recall and speed for a wide range of workloads, at the cost of using more memory because of all the stored connections.

The second family is based on clustering, commonly called IVF (inverted file index). The space of vectors is divided into clusters ahead of time (for example using k-means). At query time, the search first figures out which few clusters the query vector is closest to, then only compares against the vectors inside those clusters rather than the whole dataset. This is cheaper in memory than graph-based methods but can miss results that happen to sit near the edge of a cluster boundary, which is why IVF indexes are often tuned by searching a few extra clusters to recover recall.

Many systems also support quantization, which compresses each vector (for example by storing an approximation with fewer bits per number) so that more vectors fit in memory, usually at some further cost to recall. The common thread across all of these choices is the same trade-off: speed and memory against accuracy. Every vector database exposes some version of these knobs, even if the exact names differ, so it is worth testing recall and latency on your own data rather than trusting default settings or a vendor's headline numbers.

Filtering and hybrid search

Real applications rarely want a pure nearest-neighbour search over the whole collection. A support ticket search might need to restrict results to one customer's tickets. A product catalogue might need to restrict results to items currently in stock. This is called metadata filtering, and it matters because combining an approximate index with a filter is not always straightforward: a naive implementation might find the nearest 10 vectors first and then discover that none of them pass the filter, returning too few results. Good implementations push the filter into the search itself, but how well this is supported, and how it affects speed, differs across products, so it is worth testing with your actual filters rather than assuming it works well by default.

Hybrid search is a related idea: combining vector similarity with traditional keyword search (often using an algorithm such as BM25), then merging or reranking the two result sets. This matters because embeddings are good at matching meaning but can be weak on exact tokens that carry little semantic content on their own, such as product codes, part numbers, or exact names. If your use case depends on matching an exact identifier as well as meaning, pure vector search alone is often not enough, and you should check whether a candidate database supports hybrid search natively or requires you to combine two separate systems yourself.

Where this fits inside a RAG pipeline

In a typical retrieval-augmented generation pipeline, documents are split into chunks (for example a few hundred words each), each chunk is passed through an embedding model to produce a vector, and the vector is stored in the vector database together with metadata such as the source document, section title, and permissions. When a user asks a question, the question itself is embedded with the same model, the vector database returns the most similar chunks, and those chunks are inserted into the prompt sent to a language model, which then answers using that retrieved context instead of relying purely on what it memorised during training.

The vector database's job in this pipeline is narrow but important: store the chunk vectors, answer nearest-neighbour queries quickly, apply any metadata filters, and let you add, update, or delete chunks as the underlying documents change. It does not generate text, and it does not understand the content; it is purely responsible for the search step.

It helps to put real numbers on this to judge scale. Suppose you have 50,000 document chunks, each embedded into a 768-dimensional vector, with each number stored as a 32-bit float (4 bytes). Each vector then takes 768 times 4, which is 3,072 bytes, so the raw vectors alone take roughly 50,000 times 3,072 bytes, which is about 154 megabytes. An index such as HNSW adds extra memory for the graph connections between vectors, and as a rough rule of thumb it is sensible to budget noticeably more than the raw vector size, perhaps double, though the exact overhead depends on the implementation, the configuration, and whether quantization is used, so this should be treated as a starting estimate to benchmark rather than a guaranteed figure. At 50,000 chunks the whole collection, index included, comfortably fits in memory on an ordinary laptop or a small server, which is a useful sanity check before assuming you need a large distributed system.

The decision criteria that actually matter

Once the underlying ideas are clear, choosing a vector database becomes a matter of answering a short list of questions about your own project. These questions matter far more than any benchmark chart, because benchmarks are run on someone else's data and hardware, with configuration choices tuned for that specific comparison.

• Scale: how many vectors do you have now, and how many will you have in a year? Tens of thousands behaves very differently from tens of millions.
• Latency: does the search sit in an interactive user-facing path where a response is expected in well under a second, or is it part of a batch job where a few seconds is fine?
• Filtering needs: do queries need to be restricted by user, tenant, date, category, or permission, and how complex are those filters?
• Hybrid search: do you need to combine exact keyword matching with semantic similarity, for example for product codes, names, or short exact phrases?
• Update pattern: is your data mostly static after an initial load, or does it change constantly, with frequent inserts, updates, and deletes?
• Operational overhead: do you want to run and maintain a database yourself, or do you prefer a managed service that handles scaling, backups, and upgrades?
• Ecosystem fit: does the product have a client library for your programming language, and does it integrate cleanly with the other tools in your stack, such as an existing retrieval framework?
• Cost model: for self-hosted options this means compute and storage cost; for managed options it means understanding how pricing scales with vector count, dimensions, and query volume.
• Data residency and security: do you have requirements about where data is stored or how access is controlled, which might rule out certain managed services?

Notice that none of these questions can be answered by reading a comparison table of raw search speed. They require you to actually know your own data and your own constraints, which is why the most useful first step in any project is writing these answers down before looking at any product.

A landscape view, by category rather than by ranking

Products in this space change quickly, and specific feature lists, pricing, and performance characteristics can shift between versions, so treat the following as a map of categories to help you reason about trade-offs, and always check current documentation for the specifics of whichever product you are evaluating.

Lightweight libraries and embedded stores

Tools such as FAISS (a library rather than a full database, originally developed at Meta for similarity search research), Annoy, and embedded options like Chroma are well suited to prototyping, notebooks, and small to medium datasets. They are easy to install, run inside your own process without a separate server to manage, and let you iterate quickly while you are still figuring out your chunking strategy or embedding model. Their trade-off is that they generally offer fewer production features out of the box, such as built-in replication, access control, or a hosted network API, so moving from a prototype to a multi-user production service often means adding infrastructure around them yourself or migrating to a dedicated server-based option.

Dedicated vector database servers

Products such as Milvus, Qdrant, Weaviate, and Pinecone are built from the ground up as servers whose entire purpose is vector search at scale, usually offering HNSW or similar indexes, metadata filtering, hybrid search support of varying depth, replication, and APIs over HTTP or gRPC. Some of these can be self-hosted (Milvus and Qdrant have open-source self-hosted options, for example), while others are offered primarily or exclusively as managed cloud services. If you expect to scale into the millions of vectors, need strong filtering, and want a system designed specifically for this workload rather than adapted from something else, this category is the natural starting point to evaluate.

Vector search added to existing databases

If you already operate a relational or document database, it is often worth checking whether it has added vector search support, since this avoids introducing an entirely new system into your stack. Postgres has the pgvector extension, which adds a vector column type and nearest-neighbour indexing directly inside Postgres. Elasticsearch and OpenSearch support vector fields alongside their traditional text search, which is a natural fit if you already rely on them for keyword search and want hybrid search without a second system. Redis also offers vector search capabilities as part of its broader feature set. The appeal of this category is operational simplicity, since you are not adding a new piece of infrastructure to monitor, back up, and secure separately, and you can often combine vector search with the transactional or relational features you already depend on. The trade-off is that these bolted-on implementations may not scale as far, or offer as many tuning options, as a database purpose-built for vector search once your dataset grows very large.

Fully managed services

Managed offerings, whether a vector-specific service like Pinecone or the managed cloud version of a database such as Milvus or Weaviate, remove most of the operational burden: you do not provision servers, manage upgrades, or plan for replication yourself. This is attractive for small teams or for projects where engineering time is the scarcest resource. The trade-offs are cost at scale, less control over tuning and hardware, and a degree of dependency on the provider's roadmap and pricing changes, so it is worth reading the current pricing model carefully rather than assuming it will behave like a fixed monthly fee as your data grows.

Working through a realistic example

Consider a small company building an internal RAG assistant over 50,000 chunks drawn from its own documentation, policies, and past support tickets, queried by at most a few dozen employees at a time, with new documents added every day but nothing deleted in bulk. Walking through the criteria above: the scale is modest and comfortably fits in memory even with index overhead, as the earlier calculation showed. Latency matters somewhat, since employees are waiting on an answer, but a few hundred milliseconds of search time is not the bottleneck compared to the language model's own response time. Filtering matters because some documents are restricted to certain departments. Hybrid search would help because support tickets sometimes reference exact order numbers that a pure embedding match could blur together with similar-sounding numbers. Updates happen daily but are not high volume. The team is small and does not want to run a distributed cluster.

Given this, a reasonable starting choice is either a self-hosted dedicated vector database with good filtering and hybrid search support, run as a single modest server, or a vector extension added to a Postgres database the company already runs for other purposes, if that extension's filtering and hybrid capabilities are good enough for the use case. A fully managed cloud vector service is not strictly necessary at this scale, though it remains a reasonable choice if the team strongly prefers to avoid any self-hosting. What would not make sense here is reaching for a system designed around sharding and distributing hundreds of millions of vectors across many machines, since that adds operational complexity the project does not need yet. The right move if requirements change, for example if the company later ingests millions of documents, is to re-evaluate then rather than over-provision for a future that may not arrive in that shape.

Common mistakes

• Reaching for a dedicated vector database before checking whether a simple brute-force search, like the numpy example earlier, is already fast enough for your actual data size.
• Picking a distance metric that does not match what the embedding model was trained and evaluated with, which quietly degrades ranking quality without raising an obvious error.
• Underestimating metadata filtering requirements until the project is already in production, then discovering the chosen index does not combine filters with approximate search efficiently.
• Treating approximate search as if it always returns the true best matches; recall is typically well under 100 percent unless you tune the index aggressively, and that tuning trades away speed.
• Forgetting that switching embedding models usually changes the vector dimension or the geometry of the space, which means existing vectors generally need to be re-embedded and re-indexed, not just dropped into the same collection.
• Ignoring hybrid search when exact terms, codes, or names matter, then being surprised that pure semantic search misses queries containing a specific identifier.
• Choosing a managed service based on a quoted price without checking how the pricing scales with the number of vectors, their dimensionality, and query volume as the project grows.
• Benchmarking a candidate database only with default settings and small synthetic data, rather than with your real chunk sizes, real query patterns, and real filters.

Summary and what comes next

A vector database stores embeddings and metadata and answers nearest-neighbour queries quickly using an approximate index such as HNSW or an IVF-style clustering approach, trading a small amount of accuracy for a large amount of speed. Choosing one is mostly a matter of being honest about your scale, latency needs, filtering requirements, update pattern, and appetite for operational work, then matching those answers to one of a few broad categories: lightweight libraries for prototyping, dedicated vector database servers for production-scale workloads, vector search bolted onto a database you already run, or a fully managed service that trades cost and control for convenience. None of these categories is universally correct, and the right answer for a 50,000-chunk internal tool is often different from the right answer for a consumer product serving millions of users.

If you are building your first retrieval system, a practical next step is to prototype with the simplest option available, such as a brute-force numpy search or an embedded library, measure actual latency and recall on your real data and real queries, and only move to a more capable system once you can point to a specific limit, such as memory pressure, filtering gaps, or latency, that the simple approach has actually hit. That evidence-driven path will tell you far more about which vector database fits your project than any general comparison, including this one.