AI must never do the arithmetic

A pitch is spreading fast right now: "AI does your accounting." It sounds appealing, and it is wrong in exactly the most expensive place.

A language model produces text that is probable. It does not add — it predicts a sequence of characters that looks like a correct addition. For a sales email, that distinction is harmless. For a journal entry, a stock count, a payroll run, that distinction is the entire problem.

At XBuddy we draw a line we don't negotiate on:

The system of record does the arithmetic. AI reads, explains, drafts and warns.

This post explains why the line sits there, and why the line itself — not a stronger model — is what lets you switch AI on over real data.

Three numbers that must never be guessed

Three classes of number share a property: a small error does not announce itself, and cannot be repaired later.

Money. An invoice with the wrong tax rate is still a structurally valid invoice: it has a number, a date, a signature. It is simply wrong. And it flows into the ledger, the filing, the reports — each step adding another confident-looking derivative.

Goods. Receive 2 cartons of 24 cans and record 2 into stock, and the number is off by a factor of twelve. Nothing errors. It just means that six months later somebody is standing in a warehouse, counting by hand, unable to explain why the book and the shelf disagree.

Authority. "How much can this person approve?" is a binary question. There is no approximately correct version of it.

For these three, a system of record must produce the same answer for the same input, every time. That is precisely the property a probabilistic model does not have.

So what does AI do?

The answer isn't "less". It's different — and the different part turns out to be where humans spend most of their time.

AI reads what a query cannot

"Anything unusual in receivables this month?" is not a SQL statement. It is ten queries, plus the judgement to know which result is worth mentioning. An assistant that can read structured data — along with the semantics of each column, each unit, each currency — answers that in seconds, and cites the documents so you can check it yourself.

AI drafts; a person decides

The assistant proposes: "chase these three customers, this amount, this template" — with its reasoning and its confidence. It does not send. A person clicks Approve, and only then does the system check that person's permissions and execute through the same business service the UI uses.

That is the crux: a write goes down the same path as a manual action. Same permissions, same approval gate, same journal entry, same audit trail. AI gets no back door.

AI warns before a number becomes fact

This is the most undervalued part. An assistant that sweeps the ledger nightly and says "this bill disagrees with the goods receipt on line four" creates far more value than one that silently fixes line four — because the first preserves the chain of responsibility and the second erases it.

"But the models will get better"

They will. It doesn't change the conclusion, because the issue is not accuracy — it is verifiability.

When an auditor asks "where did this 1,842,000 come from?", the answer has to be a chain: invoice → line item → tax rate → journal entry. Traceable, reproducible, explainable to someone who does not use your software.

"The model produced it" is not an answer — even when it happens to be right. And a model that is 99.9% accurate across ten thousand rows still leaves ten wrong rows that nobody can identify.

What the line looks like in the product

Saying it is easy. What makes it real are the specific constraints:

• Assistants have no private write path. Every data change goes through a business service — the same one the button in the UI calls.
• Propose by default, not execute. A company must deliberately enable each class of action. There is no "full autonomy" shipped on.
• Some actions never auto-execute, at any setting. Emailing a customer, committing the company contractually, moving money. They run only when a named person clicks Approve.
• Amounts, quantities, units and currencies are flagged "never guess". When the information is missing the assistant asks, rather than filling a blank to look complete.
• Every run leaves a trace: who triggered it, what it read, what it proposed, who approved, what happened.

The last constraint is the one we hold hardest: if an action can't be recorded, it isn't allowed to run.

What the opposite costs

An assistant that computes and then writes produces a failure mode that is very hard to treat: wrong data looks exactly like right data. No exception, no warning, no log line. It simply sits there, and every downstream report inherits it.

The real cost isn't the correction. It's the loss of trust: after one incident like that, your finance team will manually re-check every number AI touched — and you have just paid for a tool that added work.

In closing

AI has a solid place in a business: it reads faster than a person, never tires, misses nothing, explains itself in plain language, and drafts the repetitive work.

What it does not get is the final say on a number.

That is not a temporary limit awaiting a better model. It is the correct division of labour — and the condition under which you can point AI at real data and still sleep.