- If AB testing is faster at getting results
than a usability test is,
then should user research teams change their strategy?
Maybe even do less user research?
During our September virtual UX conference,
Dr. Jakob Nielsen shares his thoughts on this question.
- It's an interesting tension here
between the sort of almost immediate feedback you get
from AB testing will only immediate
if you have a large user pool obviously,
but if you have a lot of traffic,
then you can pretty quickly get an AB test result.
And some people will say, well, that's faster.
We can just put it up there and two days later,
we know as opposed to it may take two weeks
to do user research.
But to me, I don't actually really believe
in this argument.
For first of all,
it doesn't have to take two weeks to do user research.
Now it probably does,
if you're planning a study, doing a study
analyzing a study, producing the report.
However, you can also have continuous user research,
always ongoing.
This is actually one of the things I really like
to recommend that very few people do,
but if it's Wednesdays, it's user day
and it's like every Wednesday you have four people
or five people come in
or you have connected and set up appointments
to go and have an online session with them, whenever.
You don't know in advance, what you're gonna be testing.
But one or two days before,
you pull your team
and what are your questions right now?
What would you like to get some user data on?
And that's you were to test that Wednesday?
It could be Thursday whatever day it is,
but just continuous ongoing pre-planned.
And that way you're not gonna have to wait.
You can get to your use data right away
when you have a question.
So that's one thing.
Usability studies connect to be done at almost any speed,
if you plan for it.
It doesn't have to be slow.
But the second one is I think is a more serious problem
which is that AB testing is really in many ways
a rather weak methodology for gaining insights.
Sometimes you get an answer
but you don't get the much of the why.
So you have to design variations right.
And one goes just a little bit higher than the other,
the one that does higher is the better one,
but that's only within those exact to design variations.
And so if you don't know why
then you haven't learned some lessons that can apply
to the next screen you're designing
or the next feature you're designing
or the next product you're designing.
And so that is extra information you gain
from user research where you're observing people's behavior.
So I do feel that is much more valid from anything.
There are sometimes there are just,
small subtle design distinctions that we couldn't
if by observing users tell if one is better than the other
but one is half a percent better than the other.
But it's just so little small difference
that we can tell with five users
or even with 20 users we couldn't tell.
And so we would have to run thousands of users,
but then that's exactly what AB testing is good for.
So two different shades of blue,
I think was a classic example of that,
or which one is better.
Well,if you put them up there
and then you can see a half a percent different.
You can tell that after you've gotten
a several thousand users on it
but if you pick a website you get that pretty quickly.
So, AB testing it definitely has a lot
of advantages as well,
but I don't think
that the fact that we have this methodology available
and it's quite easy with cloud based or internet based
or web based services is quite easy to get A/B data.
That doesn't mean that that's what
we should really mainly rely on
to drive our design decisions
because you get variety week insight into why?
Also of course the entire word implies
a name implies that you have to have generated A and B.
So how you gonna generate A and B?
You can sit a guest sitting around the meeting table
and guess for ideas that you might like to design,
or might to try.
But first of all, hopefully you can get more than two ideas.
And so now all of a sudden you're talking
about the multi-variant very complex test
that can take a long time to get significant data on.
But secondary gain,
that's all based on your own thinking,
like to come up with two or three variants
that you want to test.
And so this is a game where I strongly prefer
that you are not just driven by your personal intuition
about what would be interesting to test
but by insights derived from the actual customers
on things that they find easy or difficult or interesting
or exciting or whatever type of goal you have.
But by watching these people
that really drives a design inspiration,
many people think usability testing is just
for a debugging of,
I have a design let me just like smooth over
the rough edges,
and that's certainly one of the things you do,
but it's just as much
for your own good to drive your own inspiration,
inspiration derived from customers or users,
not inspiration just derived from yourself just sitting
and thinking and thinking on yourself.
Right, but thinking based on users.