So let's do an independent samples t-test in JASP. For this example, we
will be using the PuppyTrainingIndep.csv data set. We will use
this same data - in a different structure - later when we do a paired samples t-test.
Here is the setup for our research: there are two popular ways to train dogs. One
involves using a clicker, which creates a sharp click that reinforces when the dog
has done something right. The other method, of course, is food training or
treats, in which the dog gets a small morsel when he responds correctly.
Because this is an independent samples t-test, the researcher is going to use
different dogs for each group. Mickey is a dog trainer studying reinforcement
effectiveness. He assigns sixteen puppies to two different reinforcement training
techniques. After training the puppies, he measures the number of correct responses
by each puppy in ten exercises. Was there a difference between the two training
techniques? Let's walk through the five steps of hypothesis testing.
Step 1: Select the appropriate statistic. We will compare two independent samples.
The first sample is the puppies who were trained using the clicker training and
the second sample is the sample who are trained with food reward. To compare
these two samples, we will use an independent samples t-test.
Step 2: State the null and alternative hypothesis. We want to know if the means of the two
samples of puppies are different. We are not saying which group will be better, so
this is a non directional (or two-tailed) test. If we said that one group was
better, it would be a one-tailed (or directional) test. Because this is a
two-tailed test, our null hypothesis will be H0: mu 1 = mu 2.
That means that our alternative hypothesis will be H1 : mu 1 does
not equal mu 2. Step 3: Select a level of significance. We have already established
that this is a two-tailed test. Our alpha level is going to be .05.
The degrees of freedom for an independent samples t-test is (n1 - 1) + (n2 - 1)
or (n1 + n2) - 2. The total n of both groups minus 2.
Group 1 has eight puppies. Group 2 has eight puppies. So together, they have
16 puppies... minus 2 is 14. The critical value for a two tailed test,
alpha of 0.05, 14 degrees of freedom, is positive or negative 2.145.
Any t- values exceeding 2.145 indicate that the sample
of puppies being trained with clicker training was statistically significantly
different than the puppies being trained with food reward training.
Step 4: Calculate the statistics. Let's go to JASP
Begin by placing the puppy training data on your Desktop. We will open our dataset
by going to our main menu icon -> Open -> Computer -> Desktop -> PuppyTrainingIndep.csv
and then clicking Open. Let's begin by looking at our data.
The first of the two variables we will see is "Correct", which is the number of tasks
correct out of 10. This was imported as an ordinal variable, but we want to treat
it as Scale data. On the left edge of the box we click on "click here to change
column type" and set it to Scale. The second variable is "Group", which is at
the Nominal level, indicating that it defines the two groups. I don't need to
change the column type, but if you move the cursor to the right, we can "click
here to change labels". The values are 1 & 2. For value 1, let's assign the name "Clicker training",
and for value 2, we will assign "Food Reward."
We are now ready to do the analysis. Currently, I have six basic
analysis options available. I could add more modules with the plus sign, but I
don't need to do that now. The basic modules are exactly what I need.
Select T-Tests -> Independent Samples T-Test. Move the scale variable "Correct"
into the variables window. Now place "Groups" into grouping variable
JASP does the t-test and gives me the results in the results pane. This is a good start.
I can already tell that the t-test is non- significant, but I want to know much more.
All I need to do is select the options that I prefer. I am using the popular,
old-school, Student's t-test; however, the Welch's t-test is slowly
becoming the preferred t-test, because of how it is not affected by variances in
the groups that are not equal. The Mann-Whitney is a nonparametric
alternative test. We don't need that. You may recall from Step 2 that our
alternative hypothesis is that the means of the two groups are not equal. We could
also choose a one-tailed test option here if we wanted to. I would, however,
like to establish that the assumptions of the test have been met.
The two assumptions of the independent samples t-test are normality and equality of variances.
I can check the assumption of normality with a Shapiro Wilk test and
it should be non-significant, indicating that the distributions are not different
than a normal curve. The second assumption is that the two samples have
approximately equal variances, and this assumption is tested using Levine's test
for Equality of Variances. This assumption is foundational to the
independent samples t-test. It's called "homogeneity of variance".
Let me break that down: "homo" means same, and "geneity" refers to the nature of something,
in this case, the "something" is the variance (or the variability) in the data
And remember that the square root of the variance is the standard deviation.
So, homogeneity of variance means that the variances in the groups
is "of the same nature", or they vary in the same way, or they have the "same kind"
of variance. Levene's is testing whether the variances are equal (or rather,
approximately equal)... they do not have to be exact, just close enough.
That is the null hypothesis for Levine's: no significant difference between these two variances.
If the p-value is greater than .05, then the Levine's test
is non-significant, the variances do not differ, and the assumption has been met.
That is what we want. If the Levine's significance test is less than .05,
then the Levine's test is significant, the variances differ, or are
not homogeneous, and the assumption has been violated. So, in our ideal world, we
would not want variances to be statistically significantly different.
We do not want the Levine's test to be less than .05, and in later
examples, I'm going to explain why .05 may not be the best standard to use.
We would probably use . 01, or even .001
In fact, the assumption of this homogeneity of variances is the reason
why we prefer large sample sizes: because even if the variances are not
homogeneous, the t-test is ROBUST to violations of the homogeneity assumption,
meaning that we can still trust the results of the tests, but only if the
group's sample sizes are larger than 30, and the group sizes are equal.
We check in the results pane and we see that Levine's F value is .503, and the
probability associated with that F value is 0.49. Using the analogy of a dollar,
this significance value is 49 cents and that is not less than a nickel (.05).
So, is Levine's test significant p < .05 or not?
This Levine's test is
not significant. The variances are not statistically significantly different,
so the variances can be assumed to be equal - or close enough to equal - the assumption
of homogeneity of variances has been met. There is also a second assumption for
independent samples t-tests: normality. This is often understood to mean that
your dependent variable must be normally distributed; in fact, the actual
assumption is that your sample is drawn from a population of samples, in which
the distribution of sample means is normally distributed. I'm going to
explain this more fully in a lecture on the Central Limit Theorem, but for now,
a safe bet is that when you have more than 30 participants in each group, you have
likely satisfied the assumption of normality. What we need to know is
that we will test this assumption using a Shapiro-Wilks test, which, like the
Levine's test, we would prefer to be non- significant, that our sample does not
differ from normality. Under Additional Statistics, I am going to get a Location
Parameter with 95% confidence intervals. I want a Cohen's d effect size,
descriptive statistics, and a descriptives Plot. In the Results pane, we can see
the group statistics. I want to begin by scrolling to the bottom. I always like to
see a picture of the data before I begin interpreting. I see that the food reward
group mean was lower than the clicker training group, but there's a lot of
overlap in those error bars. I probably would not include this plot in my
write-up or in my paper, but it is a good place to begin. I can also see the
descriptive statistics for each group. The food group was more than a full
point lower than the clicker training group. This will be important to report
later. With these thoughts in mind, let's scroll back to the top. The Results from
Student's t-test and Welch's t-test are pretty much the same. That is because the
assumption of homogeneity of variances has been met. The more the group
variances differ, the greater the differences you would see between
Welch's and Student's t-Tests. It is always safer to trust Welch's when
you're not sure. We have our confidence intervals and, out here to the right, we
see an effect size. d = .79 is actually a pretty large effect, so we may want to
explore why the test does not appear to be significant. We have the results for
the assumptions tests right here. We can use those for our write-up.
So how can we interpret these findings?
First, the t-test value is 1.587, which does not exceed the critical value of 2.145.
The degrees of freedom is 14 (or 13.45, if you're reporting Welch's)
We can see the significance value for the t-test is 0.135,
which is greater than .05, and we also see that the 95% confidence
interval for the difference between sample means (which is m1 - m2) shows
a lower bound of negative 0.44 and an upper bound of positive 2.94.
Because this confidence interval crosses 0, it includes the null
hypothesis value of zero difference. All three of these findings tell us the same
thing: the means of these two training groups do NOT differ statistically significantly.
In addition, we see that the Cohen's d of 0.79 is actually a
large effect. The problem here is our sample size: it's so small. We did not
have sufficient power in this test to detect even this large mean difference.
Maybe we could rework our research design and use a more powerful test...
...and we will... with a repeated measures test. So for now, let's go back to our five steps
of hypothesis testing. Step 4: Calculate the statistics. The t-test value is 1.587
We will round that to 1.59.
The p-value was .135.
Using the analogy of a dollar, this p-value is 13.5 cents,
which is not less than a nickel (.05). This means that the
t-test is NOT significant and the groups do not differ
Finally, Step 5: Make the decision.
The clicker training group was not statistically
significantly different than the food reward group, t with 14 degrees of
freedom is 1.59, probability is a .135, non-significant. And here's an example
of how we would write up these results in APA style:
First, I would describe my research design and how I conducted the
experiment. Then, I would describe my assumptions checks, both for normality
and homogeneity of variances. I have two values for the Shapiro Wilks test, so I
just reported the smallest value. All probabilities were larger than 0.43
Then I would report the findings from my t-test, including the confidence interval,
and the effect size. I would also include the means and standard deviations for
each group, and those come from the descriptive statistics.
The last statement about the meaning of the findings would go in your Discussion
section, if you are writing this up for a paper. We discovered a large effect size,
as indicated by our Cohen's d of 0.79, but still the t-test was not significant.
Why not?!?
We have only eight puppies in each group: 16 total,
which is a very small sample size. This independent samples t-test
does not have much power. Like using a low-powered magnifying glass, the sample
size does not give us adequate power to detect even a large effect size.
In an upcoming example, we're going to rework our research design and use a paired
samples t-test, which has the power to magnify the effect size, like a
microscope, with more power making it easier to see. To make this interesting,
I am going to use exactly the same data points in
that test as I did in this one. The group means will be the same in both tests and
yet we will see a different outcome. Because independent measures are less
powerful than repeated or paired measures, the paired samples test is more
powerful. It is better able to find differences that exist, even in small samples.
This does not mean that there is anything wrong with using an independent
samples t-test; only that you should consider the power of your test when you
do your research design.