When you want to compare several user interfaces or user groups in a single study
there are two ways of designing the study:
Between Subjects and Within Subjects.
What do these terms mean?
Let's assume that we wanted to compare the car rental experience on two websites,
say, Enterprise and Hertz.
We can design our study in two different ways, both perfectly okay.
In one study design I could have each participant rent a car on only one site:
either Enterprise or Hertz.
This is called a between-subjects study design.
Different people test each condition so that each person is only exposed
to a single user interface.
There's another way I could design my study.
I could have every single participant test both Hertz and Enterprise.
This study design would be within-subjects.
The same person tests all the conditions.
In our case all the user interfaces.
Any type of user research that involves more than a single test condition
has to determine whether it should be
between-subjects or within-subjects.
However the distinction is particularly important for a quantitative usability study,
because it dictates the type of analysis that we can do on our data.
Note that the type of study might also impose the choice of study design.
For example, if we are interested in differences between two groups of people,
say, people under 45 versus people over 45,
or experts versus novices, we will not be able to run a within-subjects study.
A within-subjects study would require each participant to be in all conditions.
So if we're looking at age that would mean that the same person
should be both over 45 and under 45.
Or if we're looking at expertise, both an expert and a novice at the same time.
An impossible feat.
And sometimes the study itself changes the state of the participant.
For example if you want to see which of two educational websites
is more effective for teaching multiplication
you could not have the same student be exposed to both
because once she's learned how to multiply she cannot unlearn it.
A common question that people have about within-subject studies
refers to learning and transfer across conditions.
In general if you think about it after a person has completed a task on a site,
they know more about the task domain than they did before.
For example they may now know what a collision damage waiver is.
That knowledge will likely help them be more efficient on a second car rental site,
even though that second site may be very different from the first.
This can be a problem for Within study design.
It's usually addressed by randomizing the order in which
participants see the different sites, the different conditions.
In our example it means that we have to make sure
that participants don't always start with Hertz
and then move on to Enterprise.
The order of the sites needs to be random for each participant.
So what should you use,
Between or Within?
Like too often in UX the answer is
it depends on your goals and on your study.
Each type of study design has its own advantages and disadvantages.
For example, within-subject studies require fewer participants,
but between-subject studies are often easier to set up and analyze correctly
and there's no learning across conditions.