A good AI data agent says “this is not covered yet” and names what is missing. A better one turns that gap into a request the data team can review, so the next person who asks gets an approved answer.
Most writing about AI analytics measures how often an answer is right. Fewer sources ask what happens on the questions the system cannot answer. For a business reader, that edge matters more than the accuracy score.
Why is a refusal the safest failure?
The dbt Labs 2026 benchmark shows the trade clearly. On questions its semantic layer covered, accuracy was 100%. On questions it did not cover, accuracy was 0%, because the semantic layer said it could not answer. In the authors’ words, it “never returns invalid data.”
Text-to-SQL answered those same questions. Sometimes it was right. When it was wrong, the SQL ran and the number looked plausible.
A reader cannot see the difference between a correct number and a plausible wrong number. A reader can always see a refusal. That is why trust is the first concern: in Hex’s State of Data Teams 2026 report, “31% of data leaders cited trust as their top concern around AI adoption, nearly twice any other response.”
What happens after the refusal?
A refusal alone is a dead end. The business still needs the answer, so the question goes back to Slack, to a ticket, or to an analyst. Two common next steps each have a cost:
- Fall back to text-to-SQL. Anthropic’s internal analytics setup uses raw SQL “only if discovery finds no relevant metric or compile fails.” This is reasonable for analysts. For a reader who cannot check SQL, it brings the wrong-number failure back.
- Send it to the data team by hand. Nothing is lost, but nothing is learned. The same question comes back next week.
The data team’s time is already spent on these repeats. In a survey that Sigma commissioned in March 2020, “76% of data experts say between 20 – 49% of their or their team’s time is spent preparing ad hoc reports,” and 55% said a request takes between 1 and 4 weeks. The data is old, but the shape of the problem has not changed: the questions repeat.
The loop from question to approved answer
The useful next step turns each refusal into a reviewed change, so coverage grows from real demand. In Kater, the loop has these steps.
- A person asks. They ask in Claude or another MCP host, in an HTML app, or in the portal. Each command runs as that person, under their permissions.
- The agent looks for an approved answer. It searches the data team’s shared definitions and the workspace’s query contracts. A contract declares the fields it returns, the typed inputs it accepts, and the filters that always run.
- If it is covered, the agent fills the inputs and the compiler writes the SQL. The agent’s instructions say that figures come only from query results, never from its own arithmetic.
- If it is not covered, the agent says what is missing. It names the field, contract or definition that does not exist.
- The agent can propose a definition, marked as unverified. It goes in the person’s own workspace, and stays there until someone confirms it. Exploration that needs new SQL happens in the person’s Sandbox, where scratch files are personal and never versioned.
- A shared definition goes to the data team as a request. From the field picker or a definition card, Promote to a shared model files a request with a reason. An admin approves or denies it with a reason, and the person sees the decision on Home.
- The next person gets the approved answer. Once the definition is shared, every workspace that reads that model can use it.
Kater also finds repeats that nobody reported. Every night, or when an admin runs it, Kater compiles every workspace on a connection and groups fields whose formulas match. It drafts a promotion for each group and runs every affected query on the warehouse twice, as saved and as drafted. Each draft is labelled Results match, Results differ or Not verified before an admin decides. See Promotions for the details.
What should an answer show before you trust it?
“Show the SQL” is the trust signal in most product comparisons. It helps analysts. It does not help the sales lead who asked the question. An answer for a business reader should carry evidence that the reader can check without SQL.
| The answer shows | Why it matters |
|---|---|
| The definition of each number | “Revenue” can mean before or after discounts, in one currency or several |
| Every filter that was applied | Includes filters the reader did not choose, such as “paid channels only” |
| The inputs used | The country, date window or segment the answer is for |
| When the data was read | A number from last week’s run is not today’s number |
| Whether the result is complete | A truncated result cannot support a total or a ranking |
| Where it came from | An approved query, a shared definition, or exploration that nobody reviewed |
Anthropic’s internal setup adds a footer like this to every answer, with the source tier, how fresh the data is, and who owns the model. It is a good bar for any tool.
In Kater, an answer in an HTML app carries the definition of each field as the compiler renders it, and every filter the query applied, shown as a locked pill. It says when its rows were read, marks rows that are being read again, and says whether the result was cut short. Each run record keeps the actual and default inputs, the filters selected, and a snapshot of the definitions it used. A default value alone never turns on an optional filter, and a required filter always runs.
Measure coverage, not deflection
Many guides measure success as requests deflected. That rewards an answer that ends the conversation, right or wrong. These measures reward approved coverage instead. They are our recommendation, not a built-in report.
- Covered share. Of the questions people asked this month, how many did an approved query or definition answer?
- Gaps opened and closed. How many questions were refused, and how many of those now have an approved answer?
- Time to an approved answer. How long from the first refusal to the approved definition?
- Repeats still answered by hand. Which questions did an analyst answer more than twice?
A team that tracks these four numbers knows which questions to approve next, and can show the business that its questions shape the data team’s work.
Limits
- A refusal is safe only if people see it. A tool that refuses quietly, or answers “no data,” hides the gap as well as a wrong number does.
- An approved definition can still be the wrong business definition. Review by people stays necessary.
- Proposals from an agent are drafts. In Kater they stay unverified, in the person’s own workspace, until someone with the right access confirms them.