Combining SQL and Vector Search for Agent Retrieval

Why one store answers half the question

Take a realistic question an agent receives: what did we decide about retries in the payments service after the incident in March?

That question has two halves. A structured one, since payments service and after March are attributes you can filter on exactly. And a semantic one, since decided about retries will not match the wording of the document, which probably says backoff policy or delivery guarantees.

A vector store alone handles the second half and approximates the first, returning documents that are topically similar and possibly from the wrong service or the wrong year, ranked by a similarity score that has no notion of a date being outside a range.

A relational store alone handles the first half exactly and cannot do the second at all, since it can find rows about the payments service after March but cannot rank them by whether they discuss retries under a different name.

The useful system uses both, and the design question is which one goes first.

Filter first, or embed first

Filter first when the structured constraint is selective. Restrict to the payments service and the relevant date range, then run semantic search inside that candidate set. This is cheaper, because you are ranking a small set rather than a corpus, and more precise, because nothing outside the constraint can be returned no matter how similar it looks.

This is the right default for most agent workloads, since agents typically know which system, repository, project, or period they are working in.

Embed first when the structured constraint is weak or unknown. Search the whole corpus semantically, then filter the results by attributes. This costs more and preserves recall in the case where you do not yet know which subset matters, which is genuinely common in exploratory questions.

The failure mode to avoid is over-filtering. A constraint that is slightly wrong, such as a date range off by a week or a service name that changed, silently removes the answer, and the agent then reports that nothing was found. That failure is invisible, which makes it worse than returning too much, so when a filtered search returns nothing, retry without the filter before concluding the information does not exist.

Joining the results without confusing the model

Once both stores have contributed, three details decide whether the agent uses the result well.

Say what was filtered. If the search was restricted to one service and a date range, the model needs to know that, or it will present a partial answer as complete. This is a single line of context and it prevents a specific class of confidently wrong summary.

Keep provenance on every passage. Source, path, and date, attached to the text rather than mentioned separately. An agent that can cite where a claim came from produces answers a human can verify, and an answer nobody can check is worth much less than it appears.

Do not mix numbers with prose silently. If part of the answer came from a structured query, such as a count or a current value, and part came from retrieved text, label which is which. Numbers from a relational query are authoritative in a way that a number mentioned inside a document is not, and the model cannot tell the difference unless you say so.

And resolve conflicts explicitly. When a retrieved document says one thing and a structured record says another, the structured record is usually current and the document usually explains why. Presenting both, labeled, produces a better answer than silently preferring either.

Where this fits in a working setup

In practice most of the semantic half is not a database you built, it is your own material: documents, code, decisions, notes. That is where the answers to questions like the one above actually live.

RDK covers that layer. Files from local vaults, docs, and code are indexed as encrypted private chunks that agents search before querying a model, with the content unreadable to anyone else while remaining searchable by your agents. Token spend drops 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated.

Stacked retrieval describes how the halves compose in a real system: a private index over your own material answers 40 to 65 percent of queries, the public network adds 15 to 20 percent for general knowledge someone else documented well, and the model handles the remaining 5 to 10 percent. Structured queries against your operational databases sit alongside that, answering the exact and aggregate questions that semantic search should never be asked.

The division that holds: relational stores answer what is currently true, retrieval answers what is known and why, and the model handles the part where neither has an answer. Systems get into trouble when they ask one of those three to do another's job.