Hi I'm Mike Marin and in this video we'll discuss implementing a bootstrap
approach to hypothesis testing, comparing a numeric variable for two groups and
we'll do all this using R (programming language); this can be thought of as an alternative approach to
the two sample t-test, comparing independent groups. In a separate video I
explain in much more detail the concept as well as the general approach to the
test that we're going to run here. I highly recommend that you watch that
video first if you haven't already. you can find the link to that video in the
video description below. A quick reminder to subscribe and click on the bell to
receive notifications when we upload new videos. First let's import the dataset
I'm going to work without attaching the data; if we'd like to see the data in the
data viewer we can do that by using the View command (function) here, we can see that this
data consists of comparing weight gain for chicks on two different diets: casein
and meat meal, we can have a quick look at the variable names we can see that we
have both the weight as well as the feed type, the levels of the feed type casein
and meat meal, as well as how many observations there are in each of these
two feed types: 12 on casein and 11 on meat meal, so a pretty small sample size.
If we'd like to visually compare the weight gain for the two different feed
types we can use side by side box plots: here we can see visually there's a bit
of evidence that the casein may be gaining a bit more weight than meat meal
but we'd like to test this more formally. Now previously we decided to run two
different tests at the same time for the sake of demonstration, the first test statistic
that we'll calculate will be the absolute value of the difference in
the mean weight for the two diets. Now this is the same test statistic used in
a two-sided two-sample t-tests; the second test statistic will be the
absolute value of the difference in median weights for the two diets. Now at
the end of the R script used in this video we also provide code for using a
test statistic of the absolute value of the difference in the 90th percentile of
weights for the two different diets again just for the sake of looking at
using a slightly different test statistic. Now let's get to calculating
the two test statistics: the first thing we can do is calculate the mean for each
of the two different feed types: here we can ask R (programming language), to take the weight variable and
subset or remove only those whose feed type is casing and then calculate the
mean for us; we can see that is: 323.58; now we can
do the same getting the mean this time for a meat meal and I also like to
include some more advanced ways of using R in this code that you can work through
it and explore a little bit more on your own. So this can produce the exact same
results using the' with' command (function) as well as a 'tapply', then we can calculate the
test statistic 1 and we can store it in an object called test.stat1, let's do
that here and we can see the absolute value of the difference in means is
46.67 and again a way you can explore on your own makes use of the difference
command as well as with and tapply to produce the same results now we can
quickly do the exact same to calculate test statistic too we can ask R to
calculate the median weight only for those on the casein feed type as well as
the median weight for those on the meat meal feed type and again we can do that
in a slightly more advanced way making use of the with and tapply command and
let's take the absolute value of the difference in medians and store it in
something called test.stat2, here we can see our test statistic to the
absolute value of the difference in median weight for feed type casein and
feed type meat meal is 79 and again making use of the difference command we
can get that here now I won't discuss these in this video but you can have a
look at the three different options of the more classical approaches to
hypothesis testing comparing the weight of the two different feed types here you
can use the large sample two sample t-test comparing the means for the two
feed types, you can use the wilcoxon or Mann-whitney U test, this compares the
medians for the two different feed types or you could also use the
kolmogorov-smirnov two sample test, again this is comparing the distributions of
the two different feed types so let's get to talking about how to do a
bootstrapping approach here. So to do so, I'm gonna use the following
code: first I'm going to set a seed now setting a seed is not necessary but what
it does is it allows you to get the exact same set of random data each time
you run this code, okay so it's not necessary but it will allow me to say
run this code in a week's time and get the exact same results as I'm getting
now. If I don't set the seed every time I do a random resample I'll get a slightly
different set of data (in R). So I'm just doing this for reproducibility, now the next
few commands are to help me make the code a bit more general so that I can
easily take this code and change it to use on a new set of data but not have to
change very much in the code. So I'm going to use n here to set the number of
observations in the data which we saw was 23, I'm going to use B to be the
number of bootstrap resamples or bootstrap datasets I'd like to take I'm
going to start with 10,000 and you can on your own explore increasing this and
see what happens if you want and finally I'm going to set the variable to the
variable that I want to resample from so rather than asking it to resample
weights I'm just going to use variable as a more generic term again so I can
take a new set of data and I can just change the variable I'm using but not
have to change the entire set of code (in R). Now let's take a look at exactly what
this code here is doing: this code is going to generate all B equals 10,000
bootstrap resample datasets at once since a bootstrap approach resamples
with replacement we can select all the n equals 23 observations times the B
equals 10,000 observations or bootstrap samples all at once. So here what I'm
going to ask R (programming language) to do is to resample with replacement from our variable right
that is the weights and we're going to take n times B here 23 times 10,000
observations all at once, we can ask R to fill these 23 by 10,000 observations
into a matrix that has 23 rows and B equals 10,000 columns so what this is
going to do is it's going to produce a 23 by 10,000 matrix where each of the
10,000 columns is a bootstrap resample, so let's submit that code here and we
can see just like that we've already done all 10,000 bootstrap resamples.
We can check the dimensions of that to make sure things went as planned we can see
we have this matrix called bootstrap samples
that's a 23 by 10,000 matrix but again 23 rows make the rows referring to the
observations 10,000 columns each of those columns being a bootstrap sample
now what we need to do is we need to calculate the test statistic 1 and test
statistic 2 for each of the 10,000 bootstrap resamples or each of the
columns now the code I'm going to use to do this
could be made a little bit more efficient but what I want is I want the
code I'm using to be very clearly visible in exactly what the code is
trying to do it's the first thing I'm going to do is I'm going to create two
empty vectors to store the 10,000 bootstrap test statistics in one for
each of the test statistics so I'm going to create boot.test.stat1 again a vector
of 10,000 zeros, as well as boot.test.test2 another vector of 10,000 zeros now
what we're going to do is we're going to calculate the test statistic for each of
those bootstrap resamples using a loop and again this code could be made much
more efficient by writing a function and then applying this function to the
columns of the bootstrap samples matrix but I'm keeping the code more
transparent for now. We will loop through and at first for i equals 1 we will go
into the bootstrap samples matrix column i equals 1 and we're going to calculate
the mean for the first 12 observations or the mean for the 12 casein
observations and we will then subtract from this the mean of observations 13 to
23 and this is the mean for the 11 meat meal observations we will take the
absolute value of this difference and we're going to store it in boot.test.stat1
in position I equals 1 now you can see down here we're gonna do the same for
test statistic to the absolute difference in medians we then have this
repeated for I equals 2, I equals 3 all the way up to I equals B of 10,000
right for each of the 10,000 bootstrap resamples so let's submit all of that
code here and have R run through the loop calculating the 10,000 test
statistics now let's quickly remind ourselves of the observed test
statistics a reminder the absolute difference in means was 46.67 that's we
observed in our sample and the absolute value of the difference in medians was
79 now we can take a look at just the first 20 bootstrap test statistics first
four test statistic one the first 20 and then the first 24 test statistic two and
let's remind ourselves of the definition or the calculation of the p-value a
reminder that stated generically the p-value tells us under a certain set of
assumptions or a certain model what's the probability of getting our observed
test statistic or one even more extreme or larger if the null hypothesis is true
focusing on our test statistic one this is going to tell us what's the
probability of getting a test statistic greater equal to 46.67 the one we
observed if the null hypothesis were true and we'd expect the test statistic
to be roughly zero now taking a look at the first twenty bootstrap test statistics
for a test statistic one that were generated under the assumption that
the null hypothesis is true we can see that only one out of the 20 times that
48 did we actually get a bootstrap test statistic greater than the observed one
of 46.67 again recall that the p-value is calculated as the number of bootstrap
test statistics that are greater than the observed one divided by B the total
number of bootstrap test statistics so let's go through and have R calculate that.
First let's take note that we can ask R true or false questions so here
we're going to ask is the bootstrap test statistic greater than the observed test
statistic we're gonna do that here for only the first 20 so here we can see we
returned false false false false false true false false and so on so again
we're asking that question is that first one 14.7 greater than the
observed test statistic of 46.67 and we're return to false we can see the
only one were return a true for is the sixth position right which is the 48
Is that 48 greater than the observed test statistic of 46.67? yes it is, or true! now
if we ask R (programming language) for the mean of a bunch of true or falses, it will treat a false
as a value of 0 and a true as a value of 1 so let's take a look at doing that
here we're going to ask R is the bootstrap test statistic greater than
the observed one true or false and give us the mean of those, here we can you see
we are returned point 0.0832 so here we have a p-value of 8.32 percent we
can also interpret this over the 10,000 bootstrap test statistics we calculated
832 of them had test that is greater than the observed one so again taking an
interpretation of that p-value that p-value of roughly 8% what is telling us
is if there is no difference in the mean weights under the to feed types we'd see
an absolute difference in the sample means of 46.67 or more by chance roughly
8 percent of the time so again state it a bit more generically p-values tell us
what's the probability of getting the test statistics we got or more extreme
if the null is true, we can think of it what's the problem of seeing data like
we saw or more extreme if the null is true; now we can also have R go through
and calculate the p-value for test statistic to doing that we'd see it
comes out 26.3% now I'd like to bring up a reminder of
the difference between statistical significance and scientific significance
we've talked a lot about this over a lot of our videos right we don't want to get
too stuck on that magical cutoff of alpha being 5% you're in a p-value of
6.3% not being statistically significant we're supposed pvalue of 4% being
statistically significant we don't want to be too rigid in our decision
so while this difference is not statistically significant
we do have some evidence that the diets may be a little bit different and it's
worth exploring more also a reminder if we look back at these sample sizes we
only had 12 observations on casein 11 on meat meal so we really don't have that
much data to find statistical significance but we don't have evidence
that these diets may differ and worth exploring further also a reminder that
in this R script we've included some code to make a plot of the sampling
distribution of the bootstrap test statistics as well as explore a
hypothesis test comparing the absolute difference in the 90th percentile of
weight for the two diets, so you can feel free to explore this a little bit more
on your own if you want I'd also like to bring back that we've talked a lot about
how confidence intervals and not hypothesis tests may be a little bit
more useful or at least that any time when doing a hypothesis test we should
also include a confidence interval to give us an idea of how big do we think
the difference may be between these two feed types in the video to follow this
one we explore using a bootstrap approach to build a confidence interval
for the difference in means and the difference in medians using this exact
same data set so look for that to come
Thank You! and make sure to subscribe for more!