AI Analytics Agents: Why Accuracy Plateaus, and What Semantics and Memory Fix
Analytics agents plateau because the hard part is meaning, not SQL syntax. The schema does not say which revenue column is canonical, how fiscal quarters work, or which rows to exclude. A semantic layer supplies governed definitions, and curated memory of verified questions and corrections supplies the rest. Raw chat history adds little.
The bottleneck is meaning, not SQL
This topic was the subject of a recent conversation with Ashish Shubham, who is listed in his WeAreDevelopers speaker profile as a Fellow and VP of Engineering at ThoughtSpot and co-designer of its Spotter AI analyst. The analysis below is ours, written from the topic, not a summary of the episode.
A modern model writes syntactically correct SQL almost every time. That is why the failures are hard to see. Ask "what was revenue last quarter" and the agent has to decide: gross or net, bookings or recognised, calendar or fiscal quarter, including or excluding test accounts and refunds. Each choice produces a valid query and a plausible number. Only one matches what finance reports.
The schema rarely encodes those choices. Column names are ambiguous, three tables look like they hold customers, and the rule that excludes internal accounts lives in a dashboard filter someone set up years ago. A stronger model guesses more fluently. It does not guess more correctly, because the information is not in the input. That is the plateau.
What a semantic layer changes
A semantic layer defines business concepts once: metrics with their aggregation rules, approved joins, the fiscal calendar, and row-level security. The agent's job shifts from writing SQL to choosing the right governed objects and filters. Ambiguity that used to be resolved by a guess inside generation is resolved by a definition someone owns.
This is the direction BI vendors are taking. ThoughtSpot describes its Agentic Semantic Layer as the foundation for its Spotter agent, arguing in its own announcement that without one, agents misinterpret intent and miss context. Treat vendor accuracy claims as claims, but the mechanism is sound, and the same reasoning applies if you point an agent at a BI semantic model instead of raw tables.
The limit: a semantic layer covers what someone modelled. Questions outside it fall back to raw generation, and the agent should say when that happens.
Where memory helps, and where it hurts
"Give the agent memory" covers very different things, and only some of them improve accuracy.
Memory that raises accuracy
Verified question and query pairs. When an analyst confirms that a query correctly answers a business question, store the pair. Retrieving it for similar questions is the closest thing to a ground-truth example the agent can get.
Corrections to definitions. "Active customer means a purchase in the last 90 days, not a login" is a fact to capture once and reuse everywhere.
User and team context. Which region a user owns, which fiscal calendar their team uses, which dashboard they treat as the reference.
Memory that lowers it
Unreviewed conversation history is the risk. If the agent answered wrongly and nobody noticed, storing that exchange teaches it to repeat the mistake with more confidence. Memory also goes stale: a definition that changed last quarter is worse than no memory at all. Every memory write should have an owner, a date, and a way to be retired. The design questions are covered in more depth in how to design an agent memory system.
How to measure analytics agent accuracy
Build a fixed set of real business questions with the expected answer for each, agreed with whoever owns the number. Run the agent and compare result sets, not SQL text: two different queries can both be right, and a query that looks right can be wrong.
Tag each failure by cause. Wrong metric definition, wrong join, wrong time window, wrong filter, or missing permission. The distribution tells you where to invest. Mostly definition errors means the semantic layer is thin. Mostly repeated mistakes means memory is missing or unreviewed. Rerun the set on every model, prompt, or semantic layer change, the same discipline as any other agent evaluation.
Retrieval is how definitions reach the agent
Both fixes are retrieval problems underneath. The semantic layer, the glossary, and the verified query library all have to reach the agent as compact, relevant context for each question, not as a full dump of every definition. An agent that reloads the whole metric catalogue per question is slow and expensive, and it dilutes the definitions that matter.
RDK applies that pattern to private knowledge: definitions, runbooks, and verified examples are indexed as encrypted private chunks, retrieved per question, and only the misses reach the model. On repeated, reference-heavy work that cuts token spend 80 to 90 percent. The accuracy gain, though, comes from the curation, not the index.
Frequently asked questions
- Why do text-to-SQL agents plateau on accuracy?
- Because the missing information is business meaning, not SQL skill. The schema does not say which revenue figure is canonical, how the fiscal calendar works, or which records to exclude. A better model writes more fluent queries but still has to guess those choices, so accuracy stalls until definitions are supplied through a semantic layer or curated knowledge.
- Does a semantic layer make an analytics agent accurate?
- It removes the largest source of error for questions it covers, because metrics, joins, and filters are chosen from governed definitions rather than generated. It does not help with questions nobody modelled, and those fall back to raw generation. A good agent tells the user when an answer came from outside the semantic layer.
- What kind of memory should an analytics agent have?
- Curated memory: analyst-verified question and query pairs, corrected definitions, and user context such as region and fiscal calendar. Avoid storing unreviewed conversation history as truth, because an uncaught wrong answer becomes a confident repeated one. Give every memory an owner, a date, and a way to retire it when definitions change.
- What did Ashish Shubham say in the episode?
- This page does not summarise the episode, because its content could not be independently verified. Shubham is listed in his WeAreDevelopers speaker profile as a Fellow and VP of Engineering at ThoughtSpot who co-designed the Spotter AI analyst. Watch the original conversation for his views; the analysis here is written from the topic.