How to Evaluate an Agent Memory Service

Judge an agent memory service on four things: what it stores, who can read it, whether the memory moves to another agent or vendor, and what a recall costs at your volume. Underneath the branding, every one of these products is doing retrieval, so compare the retrieval, not the vocabulary.

What these products are underneath

The launches sound different and describe the same architecture. Something intercepts what happened in a session, decides what is worth keeping, writes it to storage, and searches that storage when a later query looks related. That is retrieval with a memory-shaped interface on top.

The value is real. Without it, an agent forgets everything the moment a session ends, and the user carries the burden of restating context forever. Any product that persists knowledge across sessions and pulls the right piece back at the right moment solves a genuine problem.

But treating memory as a new primitive makes evaluation harder than it needs to be. Once you see these as retrieval systems with different storage and access models, the comparison gets concrete: what goes in, who can read it, how it comes out, what it costs, and whether you can take it with you.

The four questions that decide it

Ask these before the demo impresses you.

What is stored? Some services keep raw conversation turns, some extract structured facts, some store model-written summaries. Summaries are the risky one: a model decided what mattered, the detail it dropped is gone, and you will not know what was lost until an agent confidently answers from a summary that omitted the exception.

Who can read it? Your agent memory is operational knowledge: customer specifics, internal procedures, unreleased plans. Ask whether it is encrypted at rest, who holds the keys, whether vendor staff can read it, and whether it is used to improve anyone's models. Vague answers here are an answer.

Does it move? If you switch agent frameworks next year, does the memory come with you, and in what shape? Anything you cannot export as usable text is not knowledge you own, it is knowledge you rent.

What does recall cost? Every turn may trigger a lookup. Model the fee at your real traffic, not at demo volume, and check whether recall is billed per query, per stored item, or both.

Fresh beats comprehensive

A memory store that keeps everything eventually contradicts itself: a customer's old address alongside their new one, a procedure that changed twice, a plan that was cancelled. The system then retrieves whichever chunk scores best, which may be the stale one. Ask how the service handles updates and deletion, whether newer information supersedes older automatically, and how you correct something that was stored wrong. A memory you cannot correct is a liability that grows.

Recall quality is the only benchmark that matters

Storage capacity is not the metric. What matters is whether the right chunk comes back for a real query from your domain. Test it with the awkward ones: a question phrased differently from how the knowledge was written, a topic where two stored items conflict, and a query whose answer was stored months ago. Vendor benchmarks will not tell you this, and thirty of your own queries will.

Vendor memory versus memory you own

There is a structural choice underneath the feature comparison. Memory can live inside the agent platform you use, or outside every platform in a store you control.

Inside is convenient. Setup is minimal, it is tuned for that platform's agents, and nothing needs wiring. The cost is that the knowledge is shaped by that vendor's schema and reachable through their interface, so switching platforms means either leaving it behind or performing a migration nobody wants to schedule. Adding a second agent from a different vendor means a second memory that starts empty and diverges from the first.

Outside is one more decision at setup and behaves better over time. The store is the source of truth, any agent queries it through a retrieval tool, and adopting a new tool costs you nothing in accumulated knowledge. If you expect to run more than one agent, or expect to change tools within two years, that shape wins on both counts.

The question is not which vendor has the best memory feature. It is whether your operational knowledge should be a feature of someone's product at all.

How RDK approaches the same problem

RDK treats memory as a retrieval network you own rather than a service attached to one agent. Files from your local vaults, notes, docs, code, are indexed as encrypted private chunks on the RDK network. Encryption is not an add-on tier: private chunks are encrypted, and they stay yours.

Because the index is not owned by an agent platform, any agent that can call a retrieval tool queries the same store. Claude Code reads it, a voice agent reads it, an internal service reads it. Nothing is trapped in one vendor's schema, and adopting a new tool does not mean starting from an empty memory again.

The answer to what is stored is deliberately plain: your files, chunked, not a model's summary of what it thought was important. You can read exactly what the agent will retrieve, and you correct a stale answer by editing the source.

On cost, stacked retrieval is the mechanism. A private index answers 40 to 65 percent of queries, the public network of published chunks adds another 15 to 20 percent, and the LLM handles the remaining 5 to 10 percent as fallback, so token spend drops 80 to 90 percent. Recall does not become a new bill that grows with traffic. It is what removes the larger bill you were already paying to regenerate answers.

One mechanism has no equivalent in a closed memory product. Chunks you choose to publish earn USDC per retrieval when other agents pull them, so knowledge you wrote anyway can pay for itself instead of sitting in a private store forever.

Frequently asked questions

What is an agent memory service actually doing?
Retrieval with a memory-shaped interface. It captures what happened in a session, decides what to keep, writes it to storage, and searches that storage when a later query looks related. The vocabulary differs between vendors, but the architecture does not, so compare the retrieval behavior rather than the branding.
What should I ask a vendor before adopting one?
What is stored, whether raw turns, extracted facts, or model-written summaries. Who can read it, including whether it is encrypted and whether staff or training pipelines touch it. Whether the memory exports in a usable form if you switch tools. And what a recall costs at your real traffic, not at demo volume.
Why are model-written summaries risky as memory?
Because a model chose what mattered, and the detail it dropped is unrecoverable. You will not discover the loss until an agent answers confidently from a summary that omitted the exception, the condition, or the number. Storing your actual files and retrieving the relevant chunk keeps the original text available for inspection.
Should agent memory live inside my agent platform?
Only if you expect to use one agent from one vendor indefinitely. Memory inside a platform is shaped by its schema and reachable through its interface, so switching means migrating or abandoning it, and a second agent starts from empty. A store outside every platform lets any agent query the same source of truth.
How is RDK different from a managed memory product?
The index is yours, not a feature of one agent. RDK indexes your files as encrypted private chunks that any agent can query through a retrieval tool, and what is stored is your text rather than a model's summary of it. Stacked retrieval cuts token spend 80 to 90 percent, and published chunks earn USDC per retrieval.