Why this needs its own article
An agent takes actions. A single-step AI feature produces text somebody reads.
That difference changes the risk entirely. A bad draft is discarded. A bad action has already happened.
Six risks worth understanding, and what controls each one.
1. It does something you did not intend
The main risk.
An agent misreads a situation, chooses a tool that seemed reasonable, and acts. Nothing errors. It reports success.
Why it happens: an agent chooses tools from their descriptions and from what it judges the situation to require. A misjudgement produces a confident wrong action.
The control: permissions, not instructions.
You cannot reliably instruct an agent out of a capability. Telling it “never delete records” is weaker than not giving it the ability to delete.
Practically: a dedicated account with the minimum rights the task needs. Read broadly, write narrowly, and no ability to do anything you would not want done.
2. It loops
An agent that cannot tell whether the goal is met keeps going.
Retrying the same failed step. Cycling between two actions. Working through records indefinitely.
Three controls:
A step limit. After a set number of actions, stop.
A record limit. Process at most a set number in one run.
A stop mechanism that works immediately.
Test all three before relying on it. A stop button nobody has pressed is a button that has never worked.
FIGURE 1: THE CONTROLS THAT MATTER MOST
Minimum permissions
- You cannot instruct an agent out of a capability. Remove it.
Stop before commit
- Nothing irreversible without a person.
Step and record limits
- So a loop terminates and a mistake is contained.
Full logging
- When it goes wrong, you need to see where.
3. Errors compound
A wrong step produces a wrong record, and the next step treats it as fact.
By the time anything looks odd, several things are wrong — and they are connected, so unpicking them is harder than fixing one.
The control: limit how much happens without review.
A run that prepares twenty drafts is recoverable. One that created twenty records across three systems is a cleanup exercise.
4. You cannot explain what happened
An agent takes a path, and the path varies.
When the outcome is wrong, “why did it do that” is a real question — and without a record it is unanswerable.
In a business system that matters twice over: you cannot fix what you cannot diagnose, and you cannot explain to an auditor a result you cannot trace.
The control: log every step. What it decided, which tool it called, what came back, what it did next.
Not a summary. The sequence.
5. It inherits access wholesale
There is no partial capability. An agent with the ability to post journal entries can post wrong ones.
And this compounds with the first risk. A misjudgement plus broad permissions is how a small error becomes an expensive one.
The control: treat the agent’s account like any other user account. Minimum rights. Reviewed periodically. Not an administrator.
6. Nobody reviews the output
The quiet one.
An agent producing drafts is safe only if somebody reads them.
What happens in practice: the drafts are good for three months, review becomes skimming, and then one bad one goes out.
Two controls:
Make review a real step, not a formality. Somebody with the knowledge to spot a wrong draft.
Sample deliberately, even after you trust it. A fixed proportion, checked properly.
FIGURE 2: TWO WAYS AN AGENT IS DEPLOYED
Controlled
- Dedicated account, minimum rights
- Prepares drafts, never commits
- Step and record limits set and tested
- Every action logged, output sampled
Risky
- Running with broad access
- Acts directly on systems
- No limits, no tested stop
- Drafts nobody actually reads
The control that does most work
Stop before the commit.
If an agent never does anything irreversible, most of the risk disappears.
A wrong draft is discarded. A wrong list is corrected. A wrong summary is ignored.
The value is preserved, because gathering and preparing is where the time goes — not the final click.
When you might relax it: never, for anything touching money, going to a customer, or that an auditor might examine.
Possibly, for low-consequence internal actions where the cost of being wrong is genuinely small and reversal is easy. That is a shorter list than it first appears.
Testing before you trust it
Test the failure cases, not the happy path.
- A record with missing data
- A system unavailable mid-run
- A situation the agent has not seen
- A case where the right answer is “do nothing”
- Stopping it mid-run
- Running it twice at once
That fourth one matters. An agent that always finds something to do will do something unnecessary. “Nothing needs action” has to be an acceptable outcome.
Questions before adopting one
Six.
What can it do without a person confirming? If the answer is anything meaningful, ask why.
What account does it run as, and with what rights?
How do we see what it did?
How do we stop it?
What happens when it meets something unexpected?
Who reviews the output, and will they actually read it?
Vague answers to any of these are a reason to wait.
FIGURE 3: BEFORE YOU ADOPT ONE
What can it do unconfirmed?
- If the answer is anything meaningful, ask why.
What rights does it run with?
- Not administrator. Minimum for the task.
How do we stop it?
- Test it. An untested stop has never worked.
Who actually reads the output?
- Drafts nobody reads still get used.
A proportionate view
None of this means avoid agents.
It means the controls are the design, not an afterthought. An agent with minimum permissions, a commit boundary, limits, logging and a real review step is genuinely useful.
The same agent without those is a liability that looks like a capability — and the difference does not show until the exception arrives.
The short version
Control with permissions, not instructions. If you do not want something to happen, remove the ability.
Stop before anything irreversible. That single boundary removes most of the risk while keeping most of the value.
Set and test step limits, record limits and a stop mechanism.
Log every step, because “why did it do that” is a question you will eventually ask.
And make sure somebody actually reads the output. Drafts nobody reads still get used.
Considering an agent for something that touches your systems?
Get in touch. We will go through the permissions, the commit boundary and the logging before anything is built — those decide whether it is useful or a liability.
