Why both sides matter
Most material on this subject argues one way. Vendors list benefits. Sceptics list dangers.
A business owner deciding how far to go needs both, and needs them specific enough to act on.
Here they are.
The benefits, specifically
Five, with the condition each depends on.
Less data entry. Documents read rather than typed. Depends on: clean vendor records and a review habit.
Faster answers. Questions asked rather than reports commissioned. Depends on: agreed definitions of what your terms mean.
Better-ordered attention. Which invoices to chase, which deals to call, which stock to review. Depends on: enough history for the ordering to be real.
Earlier warning. Anomalies flagged, patterns noticed, changes spotted. Depends on: tolerating false positives while it learns.
Better data quality. Because recording becomes easy, people record things. Depends on: the friction actually being removed rather than moved.
Every benefit has a condition. Vendors list the benefits; the conditions are where projects succeed or fail.
FIGURE 1: EACH BENEFIT HAS A CONDITION
Less data entry
- Depends on clean vendor records and a permanent review habit.
Faster answers
- Depends on agreed definitions, or two people get two answers.
Better-ordered attention
- Depends on enough history for the ordering to mean anything.
Better data quality
- Depends on friction being removed, not just relocated.
The risks, specifically
Six, and they are not the ones usually discussed.
1. Confident wrong answers
The main risk, and it is structural rather than a bug.
Traditional software fails visibly. AI fails fluently. A wrong figure arrives in the same tone as a right one, with no error message and no visual cue.
Mitigation: a review step wherever money moves or a customer sees the output, and defined reports for anything you must defend.
2. Bad data, amplified
AI does not introduce errors in your master data. It reaches conclusions from them faster and states them more convincingly.
Mitigation: fix duplicates, costs, units and definitions first. This is unglamorous and it is the highest-return work available.
3. Review decay
The quiet one. In the first month everything is checked carefully. By the third, output has been consistently good and checking becomes skimming.
Then one error goes through, at scale.
Mitigation: spot-check a fixed sample deliberately, permanently. Never remove review from anything published at scale.
4. Explainability
When an auditor, a regulator or a customer asks how a figure was arrived at, “the model determined it” is not sufficient.
Mitigation: use rules for anything with policy or regulatory weight. Keep AI for reading, drafting and prioritising.
5. Where your data goes
External AI services receive whatever you send them. Customer names, order details, contract terms.
Mitigation: ask directly whether data is retained and whether it trains their models. Decide deliberately, particularly for payroll, personal data and contracts.
6. Self-fulfilling patterns
Deprioritised customers get less attention, buy less, and confirm the model’s view. The prediction made itself true.
Mitigation: work a deliberate sample against the recommendation, and check calibration quarterly.
FIGURE 2: THE RISKS THAT MATTER AND THE ONES THAT DO NOT
Real risks
- Fluent wrong answers with no error message
- Bad master data amplified
- Review quietly decaying over months
- Being unable to explain a figure
Overstated worries
- That it will replace your whole team
- That it will act without being given permission
- That it is a passing fad
- That competitors are far ahead
Weighing them
The risks are real and they are almost all manageable by process rather than technology.
A review step. Clean data. A deliberate sample. Defined reports for anything external. Asking where data goes.
None of that is difficult. It is a set of habits, decided once and maintained.
The businesses that get hurt are not the ones that adopted. They are the ones that adopted and skipped the habits, usually because early results were good and the discipline felt unnecessary.
Where the balance falls
Clearly worth it, with the conditions met:
Document reading. Drafting internal text. Summarising. Ordering attention.
Worth it with care:
Customer-facing drafts, with everything read. Prediction where you genuinely have the data. Anomaly flagging, tolerating false positives.
Not worth it for most businesses yet:
Unsupervised action of any kind. Prediction on thin history. Anything you would have to defend without being able to explain.
FIGURE 3: THREE HABITS THAT MANAGE MOST OF THE RISK
Review where it counts
- Anything financial, anything a customer sees. Permanently, not temporarily.
Fix the data first
- Every risk is worse and every benefit smaller with bad master data.
Sample deliberately
- A fixed percentage checked properly, so review does not decay into skimming.
The question to ask yourself
Not “should we adopt AI?” — that is not a decision anyone can answer usefully.
“For this specific task, what does it do, how would I know if it were wrong, and what does being wrong cost?”
Three questions, per task. If you can answer all three, you can decide.
If a vendor cannot answer the first one in a sentence, that tells you something.
What to do
1. Fix your data. A week, and it improves everything regardless.
2. Write down where humans stay. Anything financial, anything customer-facing. Then hold that line.
3. Start with document reading. Easiest to verify, fastest payback.
4. Measure a baseline. Otherwise you cannot tell whether it helped.
5. Set a sampling habit and keep it after the novelty passes.
6. Decide the data question before connecting anything external.
The short version
The benefits are real and narrower than the marketing: less typing, faster answers, better-ordered attention.
The risks are real and mostly procedural: fluent wrong answers, amplified bad data, and review that quietly decays.
Every risk on that list is managed by a habit, not by a technology choice. Businesses that adopt the habits get the benefits. Businesses that skip them get the risks, usually about six months in.
Trying to decide how far to go with this?
Get in touch. We will go through it task by task — what it does, how you would know if it were wrong, and what being wrong would cost you.
