- So at the risk of offending people,
I'm gonna cover return on investment for test automation,
and I'm sure I've done that before
but I'm gonna do it again.
But this time I'm gonna give you some
exercises at the end here, which will add value
regardless of whether you think ROI is a good idea, or not.
And I'm saying that because I did those exercises
and they were useful for me.
So I wrote this blog post called "How To Invest In Testing."
Which is a parody blog post,
covers three things, ROI on test automation,
people not being resources, and 100% automation.
Gonna cover different videos in each of those
but let's look at the automation and ROI thing.
Now 'cause I'm often asked, at conferences, and in emails,
how do you calculate ROI on test automation,
and I tend not to
calculate the ROI on automation, or this particular tool,
or exploratory testing, or whatever.
Right 'cause I don't work in environments where
we have to do ROI calculations.
I don't see a lot of value in that,
I think people jump to ROI
without considering other ways of reaching decisions.
So, for me, ROI's an uneasy target,
I think that bureaucracy loves ROI
because it's a process, it's a lot, a lot of numbers,
there's calculation that's pretty
objective rather than subjective,
and there's a lot of busy work.
Every time I've been involved
in organisations doing ROI, and we've had meetings,
and we've had to work out the formula,
and we've had to work out what factors we're weighting
and everything like that.
No one, as part of that ROI process,
has factored in the cost of actually doing the ROI process,
versus the thing we are calculating ROI on.
If you've got 10 people and £500 a day
for five days, busy work, part of this ROI process,
that's £25,000, how much is the thing
that you're doing the ROI on?
And there I've just misused an ROI calculation
to try and justify not doing ROI.
But it's busy work, there's no value in there,
I didn't need an ROI calculation to figure that out.
But if you are doing ROI, factor in the cost of the process
as well to make sure that it's easily justifiable.
Consultancies love ROI 'cause they can build calculators
and they can sell you processes,
and also it means that you can't really make a decision,
so you need help, so consultancies can sell you that help.
It just so happens I've Google searched
for test automation ROI calculator
and I'm not picking on any one, but if I scroll through this
pretty much this is gonna be people who are consultancies,
have written calculators, or are telling you
how to justify buying their tool.
And that's fine, right, because that's marketing
and when we're doing marketing we have to figure
how to sell to the people that are gonna buy,
and a lot of time it's not the actual person using it,
it's the manager, or it's the manager's manager,
and they're part of a process, they're using ROI.
So I can understand why you would use an ROI calculation
in order to try and sell to those people,
'cause that's what they understand,
that's the context and environment that they're selling in.
And if you click on things for actual calculators
it's probably consultancies that are selling these things
so let's get rid of that.
Let's make sure we don't embarrass anyone
whose calculator uses these same processes.
I should really look at is,
well I don't use ROI, why is that?
Well it's easy for me 'cause I don't see any value in it
for the work that I do, and I have more autonomy
and ability to make decisions
about the work I do so I don't use it.
I don't have to use it,
I've got other ways of making decisions,
so because I don't see any value in it, I just don't use it.
I think ROI works really well on projects,
but not so much on the things within the project.
Say I was doing a huge project, like,
I don't know, HS2, High Speed 2 railway in the UK,
massive project, might want to do an ROI.
This is a report in an engineering magazine from 2011,
it's currently 2018, here it's saying
look the government could get an ROI
of between six or seven billion
on it's 13 billion investment, so essentially
the government could get a 50% return on it's investment
if it, having paid for and built HS2,
then sells the infrastructure, and that's a big ROI.
To me that's not a particularly,
I couldn't justify any decision in those terms,
I couldn't justify saying
we're gonna spend 14 billion to make something
which we'll then sell to someone else for half that price,
there's not enough in there.
What's interesting is that the ROI calculations
don't necessarily cover all the things
that are involved in that decision,
there's more going on here than just
we can sell it, we get the number back,
but ROI calculations try to simplify things.
So I'm grateful that I don't have to work
in environments where ROI is a big factor.
'Cause where's the cost of
all the homes getting knocked down,
the impact on people's lives, impact on businesses,
none of that's factored in there.
When it comes to your work, where's the factoring
of job satisfaction, is job satisfaction being better
or worse after this tool, or automation,
or whatever you're trying to invest in, comes in?
What's the opportunity cost of delay
because you're having to go through
the ROI process and calculation?
What's the difference in,
why don't we just do an experiment,
rather than calculate the ROI? Maybe we could spend the time
that we spent on ROI on the actual experiment.
It's a communication mismatch,
how is that factored into that calculation? It isn't,
so it's not necessarily the optimal way
that I would view of building a decision.
I'm not a consultancy trying to sell in big services
based on ROI, I'm not working in an environment
that forces me to do ROI,
so I have to have other ways of making decisions.
So sometimes I'll make a decision, sometimes the cost
of the actual thing we're making a decision on
is so low it doesn't actually matter.
I've just bought this,
it wasn't expensive, it was about £2.50,
I could have tried to justify it with ROI,
I could have gone on to the price charting
for New Zealand Story to see how much it sells for,
to see if that was worth it,
as an investment. I'm never gonna sell this,
I'm never gonna recoup the money for this,
but I'm gonna play it, and I'm gonna enjoy it,
so do I have to turn my enjoyment and value into investment?
That's where ROI starts to fall down for me,
sometimes I'll decide on something
that I don't have the budget,
so then I have to persuade people
to buy or give us the money for,
or increase our budget, or give us a budget.
That you could argue at that point
we're steering into ROI territory,
and maybe I have to use some sort of cost-benefit analysis,
or ROI type thing in order to do that.
But I've got other ways
that I've been able to make decisions in the past.
So if it's an environment constraint, then I might do it.
But what I'm gonna do,
is I'm gonna suggest to you an exercise.
Right because the simple fact is,
I don't really use ROI, I'm never in that circumstance,
I'm never gonna do it.
Some of you are in that circumstance,
you're in that environment, you are gonna do it.
For both of us that's our default, what's worth exploring is
is that default valid,
have we thought through the alternatives?
That's what this exercise is for, so if you're using ROI,
I'm gonna ask you to look at these questions,
why are you using ROI, what's your definition of ROI,
what do you gain, et cetera, et cetera.
I'm gonna put these questions in the
description of the video and you can work through it.
In order to keep this smallish,
I'm not gonna give you my answers,
although I have been through all those questions
and written the answers down,
and that was a really useful process for me.
So that's why I'm encouraging you to do this.
'Cause although I've made fun of it in the blog post,
I think how we make decisions and justify those decisions
is a really serious topic.
I don't think an ROI calculation, which is overly simplistic
really makes clear our thought processes
and really gives us the justification we need
to do our work, or buy our tools,
or engage in whatever activity we're doing.
Hopefully thinking through what we believe about ROI,
and about the processes we use,
can help us more than the parody post,
which the parody post is just to draw attention to
this is nonsense, what can we do instead?
Well what we can do is think seriously
about the topic of how we use decisions,
and how we justify them, and how we use numbers to do that,
and hopefully these questions will help you do that.