Let anyone in the business ask the data a question and get an answer in twenty seconds

About this series. Four pieces on what AI agents can and cannot automate in retail, written from two we built and run. Part 1 listed nine jobs a retailer can hand to an agent. This is the third of them, in detail: what it does, what it needs from you, and what it cost us to find out.

What you would be buying

A buyer, a store manager or a finance lead types a question in plain language and gets back the number, a chart, and the exact query that produced it. How has monthly revenue developed this year? Which categories are declining? What share of first-time buyers come back? About twenty seconds, no ticket, no analyst.

What it displaces is not analyst hours. It is the questions nobody bothers to ask, because the answer would arrive a week after the decision was made. That is the part that never appears on a cost line.

Running it costs roughly the price of a coffee per month at our scale, scaling to zero when nobody is asking. Getting it right is the subject of the rest of this article.

Out of the box, it answered roughly one question in ten correctly. By the end, nine in ten — using exactly the same model.

10% → 90%questions answered correctly, with no change of model
85%on a held-out question set it had never been tuned against
~20 sto an answer, with a chart and the exact SQL behind every figure
What moved the accuracy: one agreed definition per metric, automated data profiling, and daily quality gates.
None of the improvement came from the model. All of it came from the data foundation underneath.

What actually moved it

One agreed definition per metric

“Revenue”, “active customer” and “conversion rate” each mean one agreed thing, written into a business glossary the agent is required to follow. Before that, it would answer reasonably and disagree with finance — which is worse than being obviously wrong, because nobody catches it.

This is the least technical item on the list and the one that moved the most. A retailer that has never written these definitions down does not have an AI problem; it has an ambiguity problem that AI makes visible.

The platform profiles its own data

A scheduled step measures what is actually in each table — the real date range covered, the spread of each figure, the exact spelling of every category — and the agent is told those facts before it answers. So “last quarter” resolves against the dates the data genuinely holds rather than an assumption.

That habit caught a trap in our own source data: records dated a few days into the future, which would have quietly inflated every recent-period total. No prompt engineering would have found it.

Daily quality gates

Every table is checked on a schedule. When the source data broke on day one of operations, the gate caught it before a wrong number reached anyone. An agent answering confidently from broken data is the failure mode nobody plans for, because the answer looks exactly like a good one.

Privacy by construction

Customer emails, names and addresses are masked at the database level for the agent's own credentials. It is not instructed to avoid personal data; there is none for it to see, even if a user asks directly. Instructions can be talked around. Permissions cannot.

The agent itself is small

It has one tool: run a query. Before any query executes, a dry run establishes what it will cost and an unreasonable one is refused. The agent can read only approved curated tables, never raw data. Every answer carries the SQL that produced it, so an analyst can verify any figure before it reaches a board pack.

That is the whole architecture: one tool, a cost guard, a read-only and masked identity, and an answer that shows its working. The interesting engineering is in what surrounds the model, not in the model call.

How we knew it was working

Accuracy was scored against banks of business questions with verified answers — including a held-out set the system was never tuned against, where it still answered 85% correctly. Without that, “it seems good” is the only available verdict, and “it seems good” is how confidently wrong systems reach production.

We would treat a question bank as a hard requirement for any answering agent in a retail business, and as the first thing to ask a vendor for. Not a demo — a score, on questions they have not seen.

What this is worth to a retailer

  • Questions get answered while the decision is still open. The value is not analyst hours saved; it is the questions people stop bothering to ask because the answer would arrive too late.
  • The foundation is reusable. Definitions, access control, quality checks and an evaluation set are what every later agent needs. Build them once for the safest job, use them for the riskier ones.
  • It stays cheap. Ours runs for roughly the price of a coffee per month at this scale, scaling to zero when nobody is asking.

And the finding that matters most for anyone budgeting: the return came from data governance, not from a bigger model. That work is unglamorous, it is where the accuracy lives, and it pays off for every future use of your data rather than just this one.

Next in the series: the agent that acts — what changes when the agent can do something in the world, and the rule that keeps it safe. Related: our conversational analytics portfolio page, with a demo. All articles.

Start Your Data Transformation Journey

Discover practical, scalable solutions tailored to your business priorities.