I received this question from a researcher who wanted to do some group
comparisons, but the groups were of different sizes. She asked: how do I
compare two groups with different sample counts? I have two groups: the first has
40 samples and the second has 3 samples. Can you do an independent samples t-test
with such differently sized samples? COULD we do it? In the immortal words of
a certain sketchy American president, contemplating whether to obstruct
justice, "we could do it, but it would be wrong." Well then, what can we do?
Let's figure out what options we have with an unbalanced research design.
This is a circumstance called an "unbalanced research design." When you want
to compare two or more groups and those groups are of different sizes, the
research design is "unbalanced". This is true for t-tests, but also for ANOVA, and
even multivariate tests. "Unbalanced" is a problem because one of the assumptions
for the t-test is "approximately equal sample size". So we may first want to ask
"what is approximately equal?" I have taught for years that the general rule
for sample size ratio is 1.5 : 1, at the worst. Divide the larger
group by the smaller group and make sure that the ratio is less than 1.5,
and that would amount to a 60/40 split. Unfortunately, although I remember
reading about that rul,e I cannot find the original citation, so take that as
general advice, not as fact. Because assuming that the variances of each
group are equal, you CAN do a t-test on unequally sized groups. Theoretically, you
could compare a single observation to an infinite number of observations.
You COULD do it, but you might be wrong. Why does equal sample size matter in an
independent samples t-test? The primary reason is because the t-test also
assumes that the variances of the two groups are equal. This is the assumption
of "homogeneity of variances", but here's the thing: when sample sizes are equal,
then unequal variances are not a problem. The t-test is insensitive to
heteroscedasticity. We might say that students t-test is ROBUST to
heteroscedasticity when sample sizes are equal.
But as sample sizes become more unequal - the more that ratio differs from 1.0 -
then the t-test becomes increasingly sensitive to heteroscedasticity. In the
case of the n = 40 compared to the n = 3, that test will be exquisitely
sensitive to violations of homogeneity of variances, and the situation is even
worse when the smaller group has more variability than the larger group.
And that is why the t-Test results could be less than trustworthy. Now, the second
reason to consider sample size is Power. You want to maximize the power of your
t-test. You may have heard that you need at least 30 people to do a t-test, but
what if you have one group of 30 and a second group of 10? Is that a total n of
40? Or maybe you have 30 people total and you want to divide them into two groups
of 15 each? Power is computed based on the n of the smallest group; therefore,
with sample sizes of 40 and 3, your power analysis is figured based on an n of 3,
not on 40, or 43. And with an underpowered test, you are much more likely to make a
Type II error... and while we are talking about it, how low can you go? When it
comes to sample sizes, a good rule of thumb for t-tests is that each group
should have at least 6 subjects. Ideally, you should have many, many more, but treat
6 as an absolute minimum. So what are the options when we have unequal sample
sizes? Your first option - if it is not too late - is to fix your research design so
that each group is equally sized. This is true for t- tests, ANOVA, and
multivariate analysis. Get those group sizes equal. Your second option - if you've
already collected data (and this is what I would suggest for this researcher with
the n of 3) - is to collect more data. You need more people in that smaller
group, so get out there and do another round of data collection. Your third
option - if the problem is unequal sample size but you have plenty of cases - is to
randomly sample the larger group to get an equally sized comparison group.
In another video, I show you a data set with 20,000 cases, 10% of whom died.
We randomly select an equally sized group from among those who survived.
Your fourth option - if you cannot collect more data - is to use a Welch's t-test instead
of a Student's t-test. Welch's t-test does not have the assumption of
homogeneity of variances between groups, so unequal sample sizes will not affect
it, like with Student's t-test. In fact, Daniel Lakens advises that Student's
t-test is no longer the best option. Lakens' advice is in the title of his
article: Always use Welch's t-test Instead of Students t-test. So where do
you find this Welch's t-test? If you use R or JASP, then Welch's t-test is the
default test, and you can just use the results, as-is. I might also recommend
Daniel Lakens R package titled "perfect t-test" for all of your
two sample testing needs. If you use SPSS, you are already using Welch's t-test,
whether you know it or not. In any SPSS output for an independent samples t-test,
you get two lines to interpret. The top line says "equal variances assumed"
that is Students t-test. The second line says "equal variances not assumed"; that is
Welch's t-test. So the best advice is to always ignore the top line and always
report the bottom line, anytime you do a t-test in SPSS. If all goes well, then the
two outcomes will agree, but if you have problems with unequal sample sizes or
unequal variances, Welch's t-test will always be more accurate. Your final
option when things have gone wrong is to talk to your statistician. In this case, I
worked with this researcher and determined that she actually had more
data - and different data - than she thought. She simply needed to choose a different
statistical test appropriate to her data: problem solved! So here are your
takeaways: number 1 - get your group sample sizes as equal as possible.
Number 2: use the n of the smallest group for power analysis; and number 3: report
Welch's t-test, by default. And, of course, this advice only applies to independent
- or between subjects - designs in which you are comparing groups, not to repeated
measures designs, in which you are measuring the same group more than once.
This advice will assure that your t-test or ANOVA is as accurate as possible.