When running remote unmoderated studies,
we risk having participants who are not motivated
to actually try the tasks.
They just want to get their incentive money
as quickly as possible.
This can be a particularly big problem
if you're recruiting from a panel, but it can happen
even if you do your own recruiting
and you bring your own participants.
It's important to identify these people
and to make sure that you exclude their data
from your analysis.
We can think about two types of problem participants
you might want to catch: cheaters and outliers.
A cheater is a participant
who's just trying to get their incentive money and move on.
Often, these people don't even try the tasks at all.
In UX research,
an outlier is a participant whose behavior
or performance is very different
from the rest of your user population.
But just because the participant is an outlier
does not necessarily mean that they're also a cheater.
They might just not really be representative
of your user group.
So maybe they shouldn't have been recruited
in the first place.
So how do we catch these people?
The best way to do this is
to watch all of the video recordings from the study.
If you're running a qualitative study,
you should be doing that anyway for your analysis.
It isn't too hard to watch five 20-minute videos.
You should be able to notice
when a participant isn't trying the tasks.
Sometimes you'll even see people open the task instructions,
go read Facebook, or their email for a few minutes,
and then click Done to end the task.
When a person is an outlier and not a cheater,
they might be a bit harder to identify.
Listen to what they say in their video.
If you meant to recruit people with a background
in industrial engineering, for example,
and one participant seems very confused and unfamiliar
with the topic,
that tells you that maybe this is a person
who shouldn't have been recruited in the first place.
Look back at how that person ended up in the study
as a participant and double-check your screener.
If you ran a quantitative study,
you may have tested anywhere from 20 to 50 participants,
and you may not have time to watch all of the recordings.
But maybe you can just watch five minutes
in the middle of each video to spot-check.
But another quick way to find outliers
is to check the quantitative data.
Most quantitative usability studies involve
collecting time on task and task success.
With time on task,
we can look at the distribution of times
and see if there's anything that doesn't fit
within the larger pattern of data.
For example, let's look at this histogram of task times.
The majority of participants took between 10 seconds
and 178 seconds,
but we have two participants who were much faster
and four participants who were much slower.
Those people are certainly outliers for this task,
but that doesn't necessarily mean that I should go ahead
and throw away their data.
It does mean that I should investigate further.
After identifying those outliers,
I might next go and watch the video recordings
for those four participants to see what's actually going on.
Are they actually trying?
Are they actually representative of my user group?
I can also go and look at task success per participant.
So how many tasks did each participant
successfully complete?
Maybe one of these participants has a really low task time
and is failing most of the tasks.
That's a good indication that this person
is definitely an outlier and might be a cheater.
As you're looking for these cheaters and outliers,
make sure you're looking at multiple sources of data
to help you decide whether or not
they should be removed from your study.
If you do find a cheater,
check with your unmoderated platform.
Many allow you to report problematic participants
and they'll offer you a free replacement.
And if you find many outliers and cheaters,
consider changing how you recruit your users.
Keep those problematic participants
from getting in your study in the first place.