- Let's say that you've collected an important metric
for two versions of a design, A and B.
These two columns of bricks represented data
for A in the left and B in the right
with a green brick showing the mean values that we measured.
Now, if we measure something where bigger is better,
such as conversion rate,
not something where smaller is better,
like time and task,
then clearly design A is the best and should be launched.
But not so fast.
Yes, the green bricks show the means,
but there's always a margin of error
in any measurement data so let's add that.
Now the yellow bricks show the conference intervals
this means that the true value lies somewhere
within that area covered by yellow and green,
but the green isn't necessarily the exact true value.
So it could easily be the case
that the true value for A is down here,
while the true value for B could be up here.
So in that case, B would be the design we should launch.
What to do in this situation?
The statisticians will say, "We cannot make a decision
"because we don't have statistically significant data
"to prove that A is best."
However, not making a decision is also a decision
because this means that we will be keep using the old design
and lose out on any business improvements
that would result from launching a better design.
So we need to do something else than just give up.
The second answer you'll get from a statistician is that
"We could collect more data to narrow
"the margins of error for the two measurements
"in the hope of now
"having a statistically significant result.
The problem is that bigger data sets
are very expensive to collect.
Even if we're doing an AB test
with analytics that are kind of free to collect,
it's only free in terms of money.
It's a big opportunity cost
from having to wait for a long time
while the additional data is being collected.
First, we are still suffering
from using that old design instead of something better.
And second, while keeping the old AB test running,
we may be prevented from testing other design ideas
that could provide other lifts
to the business value of the product.
Let's say it costs $200,000 to collect more data.
If we're working on a website
that's worth $100,000,000 per year,
and we're expecting to gain 1%
by picking the best design option,
that means it's worth $1,000,000 to get a decision right.
So spend $200,000 to gain $1,000,000
let's do it.
But what about if we're working on a website
that's only worth $10,000,000 per year?
In that case, our hope for 1% improvement
is only worth $100,000,
but we would have to spend 200K
to gain that 100K not worth it.
In this case we have two options,
one is to go over the data anyway
even if it's not significant.
After all the mean values
are our maximum likelihood estimates
so it's more likely to not
that A is indeed the best design.
A detailed statistical analysis
will tell us exactly how much more likely this is.
But for the sake of argument,
let's just say A is 60% more likely to be the best.
So we'll launch design A at 40% risk of being wrong.
This is still better than doing nothing
because we'll usually be right
and if we're wrong, well you should not be
very wrong 'cause it's quite rare,
that it's simultaneous with the case
that A at the bottom of it's confidence level,
and that B is at the top end
of it's confidence interval.
But I actually prefer a second option,
which is to stop letting the numbers dictate the solution.
Unless the numbers are clear,
it's better for us to rely on qualitative insights.
Maybe from user testing
or maybe from heuristic evaluation
or other use of our UX expertise.
At talented UX professional
will often be able to analyze qualitative data
and form a reliable judgment as to what design is best.
If this UX judgment gives us
a strong preference for design B, then let's use B.
Strong qualitative of information
should overrule weak quantitative information.
Now, on the other hand,
if our judgment based on qualitative data
is that we can't really tell
which the sign is best,
then we're back to square one
and then the week numbers are the best we have.
But in many cases, if you have insignificant quant data,
then turn to qualitative UX insights
to drive your design decision.