RetroDeckRETRODECK

Why Your AI Agent Keeps Forgetting, and Why It Gets Slower as It Does

Most agents forget and slow down for one shared reason: they treat the prompt as memory. Each turn replays a growing history, so latency and cost rise, attention degrades as the window fills, compaction throws detail away, and nothing survives a restart. The fix is to keep the prompt small and store durable facts outside it.

Forgetting and slowing down are usually the same bug

A language model is stateless. It remembers nothing between calls, so every framework fakes continuity by resending the conversation with each request. That works for a demo. Over a long session or many sessions it fails in two directions at once: the prompt grows, so each turn costs more and takes longer, and the growing prompt makes the model worse at using what is in it.

Google Cloud Tech's AI Agent Clinic episode on this topic, published 17 November 2026, shows the pattern on a real agent. Scribant, an open source voice agent built with Google's Agent Development Kit, listens every two seconds and turns speech into calendar events, tasks and Jira tickets. According to the episode, sending the full transcript on every tick burned tokens, spiked latency, and still left the agent with no memory once the app closed. That is the whole problem in one sentence: replaying history is expensive, slow, and still not memory.

Five causes, and the symptom each one produces

Each cause leaves a different fingerprint. Match the symptom before you pick a fix, because the fixes do not overlap much.

1. Nothing persists after the session

Symptom: a new session asks questions the agent answered yesterday. The history lived only in process memory or a temporary buffer, and it died with the process. No amount of prompt tuning fixes this. You need a store that outlives the session and a step that writes to it.

2. The window filled and old turns fell off

Symptom: in a long session the agent ignores instructions from the start. Many frameworks keep the last N messages and silently drop the rest. Early constraints, the user's name, the goal of the task: whatever was said first goes first.

3. Context rot: the fact is there, but the model uses it badly

Symptom: the forgotten detail is still in the prompt when you inspect it. Chroma's July 2025 study tested 18 models and found reliability falls as input length grows, even on simple retrieval tasks, and falls faster when the relevant passage is surrounded by similar distractors. Anthropic's context engineering guidance from September 2025 describes the same effect as context rot and treats attention as a finite budget that every token draws down.

4. Compaction threw the detail away

Symptom: the agent remembers the gist but not the specifics: the ticket number, the exact limit, the edge case the user flagged. Summarizing old turns keeps the window small, but summaries keep themes and lose identifiers. If forgetting started right after the first compaction, this is the cause.

5. Full-history replay makes every turn heavier

Symptom: latency and cost climb steadily through a session. The model processes the whole input on every call, so time to first token grows with prompt length. If each turn resends everything before it, total tokens processed across a session grow roughly with the square of the turn count, not linearly. Prompt caching cuts the cost of a stable prefix, but the model still has to attend over all of it.

A four-question diagnostic

Run these against logs from one long session before changing anything.

Does the agent forget across sessions or within one? Across sessions points to cause 1. Within one points to causes 2 to 5.

Is the forgotten fact still in the prompt sent to the model? If it is missing, the window or the compactor removed it. If it is present, you have context rot, and adding more context will make it worse.

Does the prompt token count rise on every turn? Log it per call. A line that only goes up means full-history replay, and latency will follow it.

Did the problem start at a specific turn? A cliff, rather than a slope, usually marks a truncation or compaction event.

Fixes matched to causes

For no persistence: extract durable facts into a store that survives restarts, and inject only the relevant ones at the start of a session. In the AI Agent Clinic episode, the team uses Memory Bank on Google's Vertex AI to extract, deduplicate and reconcile user facts across sessions; the episode reports injecting 24 tokens of distilled memory where its naive retrieval baseline used roughly 1,400.

For truncation and rot: give each component only the context it needs. The same episode feeds Scribant's intent classifier just the latest 40 words of transcript instead of ten lines and reports 86 percent fewer classifier tokens. Those are the presenters' own figures for their agent, not a general benchmark, but the principle transfers: a narrow step does not need the whole conversation.

For compaction loss: pin the things a summary must never drop, such as identifiers, constraints and open tasks, in a structured block outside the summarized history. Anthropic's guidance calls this structured note-taking: the agent writes notes outside the window and reads them back when needed.

For replay cost: stop treating the transcript as the memory. Retrieve what the current turn needs, keep a short rolling window for conversational flow, and hand long subtasks to sub-agents with clean windows that return a summary.

Where a retrieval layer fits

Every fix above has the same shape: keep the prompt small and fetch facts on demand. That is retrieval, and it works best when the knowledge lives somewhere the agent can search before it calls the model. RDK indexes local vaults, docs and code as encrypted private chunks; agents query them over MCP, the CLI or the Partner API and get a few hundred tokens of context instead of whole documents. On repeated or reference-heavy work that cuts token spend 80 to 90 percent, and a smaller prompt is also a faster one.

Frequently asked questions

Why does an AI agent forget things in the middle of a long conversation?
Usually because older messages were dropped or summarized to fit the context window, or because the window is so full that the model uses early content poorly. Check whether the forgotten detail is still in the prompt. If it is gone, fix truncation or compaction. If it is still there, the prompt is too long and needs trimming, not extending.
Why does an AI agent get slower the longer it runs?
Most agents resend the full conversation on every call, and the model must process every input token before it answers. As history grows, time to first token and cost per turn grow with it. Across a session that resends everything, total tokens processed rise roughly with the square of the number of turns. Retrieval and rolling windows flatten that curve.
Will a bigger context window stop an agent from forgetting?
It delays truncation but does not fix the other causes. Research such as Chroma's 2025 context rot study found model reliability declines as input grows, so a bigger window filled with history can make answers worse. It also raises cost and latency per turn, and it still forgets everything when the session ends unless you add persistent storage.
What is context compaction, and why does it lose information?
Compaction replaces old conversation turns with a shorter summary so the window stays under its limit. Summaries keep the gist and drop specifics like IDs, numbers, exact wording and edge cases. To avoid losing what matters, store critical facts in a separate pinned block or an external memory store rather than trusting them to survive repeated summarization.
How do you give an AI agent memory that persists between sessions?
Add a write step that extracts durable facts, preferences and decisions into a store that outlives the process, and a read step that retrieves only the relevant items at the start of each session or turn. Managed services, a database, or a retrieval index all work. Replaying the old transcript is not persistent memory and becomes slower every session.