About this series. Four pieces on what AI agents can and cannot automate in retail, written from two we built and run. Part 1 listed nine jobs a retailer can hand to an agent. This is the third of them, in detail: what it does, what it needs from you, and what it cost us to find out.
A buyer, a store manager or a finance lead types a question in plain language and gets back the number, a chart, and the exact query that produced it. How has monthly revenue developed this year? Which categories are declining? What share of first-time buyers come back? About twenty seconds, no ticket, no analyst.
What it displaces is not analyst hours. It is the questions nobody bothers to ask, because the answer would arrive a week after the decision was made. That is the part that never appears on a cost line.
Running it costs roughly the price of a coffee per month at our scale, scaling to zero when nobody is asking. Getting it right is the subject of the rest of this article.
Out of the box, it answered roughly one question in ten correctly. By the end, nine in ten — using exactly the same model.
“Revenue”, “active customer” and “conversion rate” each mean one agreed thing, written into a business glossary the agent is required to follow. Before that, it would answer reasonably and disagree with finance — which is worse than being obviously wrong, because nobody catches it.
This is the least technical item on the list and the one that moved the most. A retailer that has never written these definitions down does not have an AI problem; it has an ambiguity problem that AI makes visible.
A scheduled step measures what is actually in each table — the real date range covered, the spread of each figure, the exact spelling of every category — and the agent is told those facts before it answers. So “last quarter” resolves against the dates the data genuinely holds rather than an assumption.
That habit caught a trap in our own source data: records dated a few days into the future, which would have quietly inflated every recent-period total. No prompt engineering would have found it.
Every table is checked on a schedule. When the source data broke on day one of operations, the gate caught it before a wrong number reached anyone. An agent answering confidently from broken data is the failure mode nobody plans for, because the answer looks exactly like a good one.
Customer emails, names and addresses are masked at the database level for the agent's own credentials. It is not instructed to avoid personal data; there is none for it to see, even if a user asks directly. Instructions can be talked around. Permissions cannot.
It has one tool: run a query. Before any query executes, a dry run establishes what it will cost and an unreasonable one is refused. The agent can read only approved curated tables, never raw data. Every answer carries the SQL that produced it, so an analyst can verify any figure before it reaches a board pack.
That is the whole architecture: one tool, a cost guard, a read-only and masked identity, and an answer that shows its working. The interesting engineering is in what surrounds the model, not in the model call.
Accuracy was scored against banks of business questions with verified answers — including a held-out set the system was never tuned against, where it still answered 85% correctly. Without that, “it seems good” is the only available verdict, and “it seems good” is how confidently wrong systems reach production.
We would treat a question bank as a hard requirement for any answering agent in a retail business, and as the first thing to ask a vendor for. Not a demo — a score, on questions they have not seen.
And the finding that matters most for anyone budgeting: the return came from data governance, not from a bigger model. That work is unglamorous, it is where the accuracy lives, and it pays off for every future use of your data rather than just this one.
Next in the series: the agent that acts — what changes when the agent can do something in the world, and the rule that keeps it safe. Related: our conversational analytics portfolio page, with a demo. All articles.
Discover practical, scalable solutions tailored to your business priorities.