Skip links

Measuring Whether Automation Worked

Why most companies cannot answer

Ask whether an automation paid off and the usual answer is “it feels faster”.

That is not an answer, and it is why marginal automations survive for years and good ones never get repeated.

The reason is almost always the same: nobody recorded what it was like before.

A baseline takes an afternoon. Without it, every claim afterwards is an impression.

The four baseline numbers

Record these while you are still watching the manual process.

Time per run. Minutes, measured, not estimated. People underestimate tasks they do often and overestimate ones they dislike.

Frequency. Runs per week or month.

Error rate. How often something goes wrong. Even roughly — “about one in twenty needs correcting”.

Delay. From trigger to completion. Not how long the work takes — how long it waits.

That fourth one is often where the real problem is, and it is the one nobody measures.

FIGURE 1: THE FOUR BASELINE NUMBERS

Time per run

  • Measured, not estimated. People misjudge familiar tasks.

Frequency

  • Runs per week or month.

Error rate

  • Even roughly. One in twenty is a usable number.

Delay

  • Trigger to completion. Often the real problem.

The savings people forget to count

Time is the obvious one. Three others are often larger.

Errors avoided

A person entering four hundred records makes mistakes. A system does the same thing the same way each time.

Value it properly. The cost of one wrong entry reaching a customer — the correction, the credit note, the relationship — can exceed a year of time saved.

Delay removed

A process that runs overnight instead of when somebody gets to it.

For order processing, stock allocation or customer response, hours matter. An order confirmed at 9am instead of 4pm is a day earlier through the warehouse.

This rarely appears in a business case and it is often the biggest operational change.

Work that starts getting done

The largest benefit, and the one nobody counts.

The reconciliation nobody had time for. The supplier review that never happened. The check that got skipped when things were busy.

Automation frequently adds work that was being quietly skipped, rather than only removing work being done.

And skipped work is where problems accumulate unnoticed.

The cost side, honestly

Three parts, and the third is the one people leave out.

Build cost. Time or money.

Running cost. Licences, per-task charges, hosting.

Maintenance. For as long as it runs.

Maintenance is not small. Screens change, APIs change, systems get upgraded, credentials expire. Somebody attends to it.

A rough guide: if the annual saving is not several times the annual cost including maintenance, it is marginal.

Marginal automations get abandoned when whoever built them leaves — and then you paid for the build and got a period of benefit.

FIGURE 2: THE HONEST CALCULATION

Count as savings

  • Time per run × frequency
  • Errors avoided, valued properly
  • Delay removed
  • Work that starts getting done

Count as costs

  • Build time or fee
  • Licences and per-task charges
  • Maintenance, for as long as it runs
  • Your own time reviewing output

Comparing afterwards

Wait long enough. A month at minimum, and long enough to have hit the exceptions.

Then compare against the same four numbers.

And ask two more questions:

How much time went into maintaining it? Include this. It is part of the result.

Where did the freed time go? If nobody can say, it dispersed — and that is worth knowing before the next project.

Watch for the trade

Faster is not automatically better.

Three things to check:

Did error rate rise? If the automation is fast and slightly wrong, that is a loss dressed as a win. Check this specifically.

Did work move rather than disappear? Sometimes automation shifts effort to a review step, and the total is similar.

Did it create a new failure mode? A manual process fails visibly. An automated one can fail quietly for weeks.

That last one is why alerting matters, and why a silent automation with no monitoring is not a saving at all.

Reviewing over time

Once a year, per automation.

Is it still running? Some quietly stopped and nobody noticed.

Is it still needed? The process may have changed.

What has it cost to maintain in the last year?

Is the benefit still there?

Switching one off is a valid outcome. An automation costing more to maintain than it saves should go. That is a good decision, not a failure — and companies that never retire anything accumulate a maintenance burden that quietly consumes the benefit of everything else.

FIGURE 3: THE MEASUREMENT CYCLE

Baseline before

  • Time, frequency, errors, delay

Build and run

  • A month minimum, through the exceptions

Compare honestly

  • Including maintenance time

Review yearly

  • Retire what no longer pays

What to report

Keep it to four lines.

Hours saved per month, net of maintenance.

Error rate, before and after.

Delay, before and after.

What the freed capacity is now doing.

That last line is what makes it real to anyone senior. “Four hours a week” is abstract. “Four hours a week now going into supplier negotiation” is a result.

The short version

Take a baseline before you build. Time, frequency, errors, delay. It takes an afternoon and everything afterwards depends on it.

Count the savings people forget — errors avoided, delay removed, and work that starts getting done.

Count maintenance as a cost, for as long as the automation runs.

And review yearly. Retiring something that no longer pays is a good decision, and doing it keeps the rest worth having.

Automations running but nobody sure whether they pay?

Get in touch. We will help you measure what you have — including which ones are worth retiring.

Leave a comment

Drag