- When we talk about quantitative usability testing,
we usually recommend that you include
a fairly large number of users, around 40, to make sure
that the numbers obtained from your study generalize well
through the whole user population.
When you include just a few participants in a study,
it's possible that their individual circumstances
and backgrounds skew the results
in one direction or another.
Now, almost inevitably,
when our conference attendees hear that,
they start questioning another one of our recommendations,
to test with only five users in qualitative user tests.
After all, if you test with only five users,
it means that your findings will not generalize
to the whole population, doesn't it?
This question is so frequent that we need to address
the misunderstanding.
The reason for different sample size recommendations
in qualitative versus quantitative studies stems
from their different goals.
In quantitative usability studies, the goal is to measure
the usability of a system by estimating
how your general population will fare on measures
such as task success, errors, satisfaction, and task time.
If we include just a few participants in the study,
the resulting numbers will be unreliable
and not capture the behavior of the general population.
In contrast, qualitative user studies
are not trying to capture the behavior
of the whole population.
Rather, they just aim to detect usability issues
with the design so these could be fixed
and the team could move on to a new, better version.
The new version will usually also get tested
and improved on.
Qualitative studies do not try to predict how many users
will complete a task, nor do they attempt to figure out
how many people will run into any specific usability issue.
And while there is a statistical uncertainty
about any number obtained from a quantitative study,
there is absolutely no uncertainty in a qualitative study.
Any error identified is a legit problem
that someone encountered and needs to be fixed.
In the early 90s, Jakob Nielsen and Tom Landauer
built a mathematical model that shows
that by doing a qualitative test with five participants,
you will identify 85% of the issues in an interface.
And NN/g has repeatedly argued, and justly so,
that a good investment is to start with five people,
find your 85% of the issues, fix them,
then test again with another five people, and so on.
It's not worth trying to find all the issues in one test
because you'll spend too much time and money,
and then you'll be sure to introduce other issues
in the redesign.
To recap, when you are collecting numbers,
you should include larger sample sizes in your study
in order to make sure that your numbers capture
the behavior of the whole population.
After all, if you only look at two to five users,
what's the chance that their average time on task
will be close to the average of your general population?
But if you are simply trying to collect usability issues,
even watching five users will give you plenty
of real issues to work on.