还需要矢量数据库吗？ RAG、AI 代理和搜索

Are vector databases still needed? I ask because there's a good deal of discussion online about folks replacing them with a text search tool from 1973 called grep. And people are claiming they're getting better results because the grep solution is a good deal simpler and also doesn't need a separate index and an index can, can go stale over time. So, do we need a vector database? Well, first of all, what is it? Essentially, a vector database is, well, like every database in that it stores content, but it stores content specifically as embeddings. And these are a series of numbers, which just are points in our high-dimensional space, and they're arranged so that things with similar meanings sit near each other. And then a retrieval time, a vector database will return items that are deemed semantically similar to what is being searched for. It's commonly used in scenarios like retrieval augmented generation to look up information to include in a prompt that gets passed to a large language model. So, let me lay out four retrieval steps as a pyramid. Now the bottom level is no retrieval at all where everything is just kind of pasted into an LLM prompt. And at the top there is a dedicated vector database. And the rule is we're only going to move up the pyramid when the workload actually warrants doing that. There's no point scampering to the top for the sake of it and taking an unnecessary fall. Okay, so why dispense with vector databases, not use them? I'm gonna propose three different reasons and the first reason is because you might just wanna do something called agentic search instead. So the alternative to a vector database is semantic search is just good old fashioned keyword search using keywords to find information. And that just matches the literal words in a query. In fact, that is keyword search exactly what grep is doing. It's a command line utility for searching text for lines that match a regular expression. And in many cases, keyword search can be a bit of a dumb tool but an autonomous AI agent using keywhy, keyword search it doesn't have to get retrieval right the first time in one shot. It can run a search, it can, can read what comes back and then maybe refine the search and just kind of keep going again and again. That's part of the agentic loop. So keyword search, even if it takes a few tries, can be as good as vector search without some of the downsides. Now another reason that vector databases just might not be needed is we could just consider that we can use a very big context window. So, long context. Now a frontier model today can take in something like a thousand times more text than it could a few years ago. So why bother retrieving at all? Just kind of throw in all the necessary information into the context window. And then the third reason really comes down to cost and specifically we're talking about the cost of creating these embeddings. So embedding costs. Now with semantic search, every time we take a document like this one, it has to be chunked up and then embedded, turned into a series of numbers. And that has to happen on the way into the database. And also every time, we issue a query, we need to embed that as well, because the question has to become a point in the space before it can find its neighbors. And if you happen to switch embedding models, well, every document would need to get embedded all over again. Maybe the workload just isn't worth all of that. But even then, there might not be a need for a dedicated system to do all of this, because pretty much every mainstream database now supports vector search just in regular databases, and embedding is after all just extra column of numbers next to the row it describes and indexing columns is exactly what databases do. So, the argument goes that if semantic search is needed, just do it in the database that's already running. So that's the case against. And these three reasons they kind of line up quite nicely with my pyramid over here. So, at the bottom level, well long context that's really considered no retrieval. And the next level up in the pyramid, that is keyword search, which might be performed by an agentic supervisor. So agentic search maps to that, because it could perform all of this without needing any embeddings. And then embedding costs fits nicely with the next layer up, which instead of having a vector database, is just to have a vector index that sits in an existing database. But are there still reasons to keep climbing the pyramid all the way to the top? I think, I think there are. Now the bottom level is basically saying paste everything that you're gonna need into a context window. So maybe we're gonna take any relevant documents that we need here and just put them directly into the context window that we're gonna send to an LLM here. Now, a larger context window does mean higher token processing costs, but I think the bigger problem might just come down to accuracy when you really fill up a context window like this. Context bloat can result in an AI model performing worse than retrieving just the relevant sections from a knowledge store when they're needed. So I think no retrieval is fine when the corpus is small but when it gets particularly large, you might wanna step up the pyramid to keyword search. But keyword search has its issues as well. I mean, let's consider a workload is an internal support assistant and a user types: how do I reset my password? Now, it turns out that there is already a ticket created to answer exactly that question and it's called credentials loop after update. Now, keyword search, strict keyword search isn't gonna find this ticket because these two things here don't share a single word. So it's all like grep is not gonna return anything. In embedding space though, these two sentences are close neighbors. This is what is known as the semantic gap. The, the user and the ticket are describing the same problem in completely different words, which is something that semantic search was built for. So yay embeddings! Maybe we have a workload that would benefit from semantic search after all. So up the pyramid we go into a vector index. Now, a vector database itself is a good solution up to a point, but well, what happens if we get to a lot of different vectors, a lot different embeddings? Well, it turns out that when we have a database here that just has an index of vectors and we fill that with some embeddings like this, each of those embeddings individually don't take very much space, maybe a few kilobytes of memory. And the index makes searching fast. But if we've gotta keep all this in memory because there's a lot of them, maybe we have millions and millions of these vectors, a general-purpose database can start to show a bit of strain. So that's the point where maybe a dedicated vector database is a better fit because vector databases are built to hold billions of vectors and still answer in milliseconds. So are vector databases still needed? I think it actually comes down ta, to two questions. And the first question is, is vector search still needed? And if the answer to that is yes, then the second question is, is a dedicated vector database still needed? Vector search itself is in nearly every database as a feature. So, for some workloads, the answer is just to search the, the vector index that came with the database already running. And for workloads with hundreds or millions of vectors, and those are getting queried constantly and need to get back quickly, then maybe a dedicated vector database is the place. So, if information fits comfortably into the context window, then maybe forget vector databases entirely and RAG and all of that stuff, and just use this whole no retrieval pipeline. If a grep keyword search can do the retrieval, that's a nice, lightweight option, and AI agents can make a lot of grep calls as they need them. But if the semantic gap, meaning the difference between keyword search terms and the information you're trying to find, might be a problem, then creating a vector index is a good idea, and then a full-on dedicated vector database makes sense when the number of vectors runs into the many millions. So, which level of the pyramid is your workload on? Let me know in the comments.