When running statistical tests to check for a difference,
we always aim to keep everything constant, except for the item being changed.
Recall the example where we were testing the feedback control system
between the existing controller, A, and the new controller,
B. We would have kept the materials constant, used the same operator and recipe
to operate the batch, and the same analytical methods to analyze the results.
If we did not do that, then those other changes, might have biased the result.
In practice, those external changes will add extra variation into the process.
And remember, from the equation for the confidence interval, a larger variation here
in the standard deviation will make the confidence interval wider than it should be.
This could lead to you incorrectly concluding the change from A
to B was not effective, when it actually was.
There are cases though where we are not able to keep things constant.
Let's use this example for the rest of this video lecture.
We are trying to test if adding baffles to our reactor will improve the conversion.
We have a testing reactor for our experiments that allows us to add and remove the baffles.
If the results show higher conversion with the baffles, then we will go install these
on the many 100's of reactors we have in our company, globally.
The problem is that the reactor is large, and each container
or raw materials will be enough for only 2 batches.
We cannot keep the raw material constant in this case.
Every second batch we will have to switch to a new container,
and we know that the raw material also affects the conversion.
One way you might get around this is to decide that you will run 10 batches,
so take 5 containers of material, combine them into a large tank, mix them up,
and then split them out into the 10 samples.
But it is likely you don't have a big enough tank for the 5 containers of raw material,
and no practical way to mix that much material either.
The way we can resolve this issue is to recognize what our goal is: we want to determine
if a DIFFERENCE exists: with baffles or without baffles.
We can run one batch with baffles, and run the next batch
without baffles using the other half of the raw material.
We will get two conversion numbers.
If there was an improvement, then the DIFFERENCE between the conversions should show that:
one value will be higher than the other.
And if the baffles lead to a consistent improvement, we should observe that in the data
for batches 3 and 4, batches 5 and 6, and so on.
This subtraction of the conversion numbers subtracts out the effect of the raw material.
If the raw material does affect the conversion by a certain amount,
then after we have subtracted the numbers, that raw material effect will cancel out.
Let's look at this mathematically: let's say the conversion from the first reactor test was 71%,
but the raw material has an effect of 3 units, so we measure a conversion of 74%.
In the reactor with baffles the conversion was 75%, but, the raw material's effect
of 3 units means that we observed a value of 78%.
When we subtract the numbers: 78 minus 74, we get a difference of 4 units.
Notice that in practice we of course do not know the effect of the raw material, that 3% bias;
all we do know is that if we subtract, it will cancel out.
And even if there was no effect from raw materials, when we subtract,
we still land up with a difference of 4 units.
This is why pairing is so effective: it removes a common, constant source of variation.
In fact, a paired test must be designed, or planned ahead of time.
We always know when to use a paired analysis: pairing is used when there is something
in common WITHIN the pairs that could affect the outcome.
There is no commonality BETWEEN the pairs.
In this example, the common element WITHIN the pairs is the raw material.
The raw material is not common BETWEEN the pairs.
Every time we run a new pair, we use a new set of raw material.
So let's go take a look at how you run these experiments.
Never run all your A samples first, followed by your B samples.
This is the worst way you can do it, because something will almost certainly also change
when you are doing the A samples, then move to the B samples.
For example, the temperature in the factory could get warmer during the day,
which will affect your outcome, or there might be a shift change and a change of operators.
Then you end up contaminating your results with that effect.
We use the term: confounding: you have confounded your results.
And we will see that term later on in the course again.
One other option is you could randomly go do your experiments: here I have used a coin
and determined to run A, A, B, B, B, A, A, B, A, then B. Then I work through my five containers
of raw materials: container 1 for the first two,
then container 2 for the next two batches, and so on.
This is OK, but the raw material effect will still contaminate your results,
and you have not cancelled it out.
We have to intentionally go set up our experiments,
and the best way to do it is as shown here.
In a spreadsheet, create a column with the thing you are testing:
no baffles vs baffles in this case.
Then add a column for the common element between your runs.
In this case, it is the raw material from a particular container.
In which order should we run these experiments?
Create a column of random numbers between any low number and high number, and then paste them
as values, that's so the spreadsheet doesn't keep changing these random numbers.
Sort your spreadsheet based on that random column, from low to high.
Now you have got the order in which you should do your experiments.
So now we go, and run the experiments,
and as each experiment is completed I will go fill in the results.
This might take several days.
Once we are completed, we go sort the data, back to how it was.
We are ready to go calculate those differences now.
The first container showed a difference of 76 minus 77; then the data from the next container
of raw materials was 78 minus 74, the next was 81 minus 83;
then 73 minus 74, and finally, 73 minus 81.
Mmmm, it looks like the baffles didn't have that great an effect,
there are quite a few negatives here, indicating that they may have decreased conversion.
But back to the statistical theory for a minute.
These 5 differences, let's call them w, are just a random variable as well.
We are interested to know if these w's are near zero,
or if they are large positive or large negatives.
And we have a tool for that already: a confidence interval.
Let's see: we calculate the average w, call it is w-bar.
From our earlier knowledge of the central limit theorem,
w-bar will come from the normal distribution.
Let's call the population mean mu-w, and the variance is sigma-w squared
over n. Remember this n will be the number of pairs we had, 5 in this example,
5 data points of w. Well mu-w can be seen to be either mu_A minus mu_B
or mu_B minus mu_A, depending how you subtracted.
What that last part implies is that mu_W will be zero if there is no difference.
In other words, our confidence interval for mu_W should span zero if there is no difference
between A and B. To calculate that confidence interval we need a z-value for w-bar.
Because w-bar comes from the normal distribution,
we can go subtract the population mean, mu_W, and divide by the standard deviation shown here.
Now please note that the standard deviation, and we will go estimate it from the data.
The key point is that the estimated standard deviation implies that the value for z now comes
from the t-distribution, between lower and upper critical values: -c_t and +c_t.
So we can unfold that z-value next to end up with the confidence interval.
I will emphasize again, that n is the number of pairs of data you have,
not the number of experiments you did.
That's because this confidence interval is for w's: the difference between the paired values.
We can go calculate this confidence interval for ourselves: calculate w-bar from the spreadsheet,
and also the standard deviation of the w's.
The critical value of t can be assumed to be
from a 95% confidence interval, so 2.5% in each tail.
Pause the video and take a few minutes and calculate
that lower and upper bound for yourself.
Do your values agree with mine here?
The lower bound is -6.9 and the upper bound was 3.7.
I used the qt(...)
function in R to get the critical t-value of 2.78, with 4 degrees of freedom.
Let's interpret those bounds: firstly, because it spans zero, it indicates to us
that there is no significant difference between baffles and no baffles.
Does that imply then we should go install baffles?
Well let's see: the bounds are asymmetrical as well:
and they skew in favour of the case of no baffles.
Remember I defined w as B minus A: with baffles minus no baffles,
and w-bar was a negative number, that indicates that adding baffles lead
to a slightly worse, lower conversion.
Even from a practical standpoint, adding baffles might just decrease conversion a little bit.
So there is no compelling reason in these 10 data points to recommend adding baffles.
Now what if baffles improved something else about the process?
Let's say, while you were running these tests you noticed the batch had better mixing,
and as a result the final product had a better appearance for the customer.
What would your recommendation be?
I would probably recommend adding baffles
if the appearance requirement was important and it had economic value.
It appears that conversion does not suffer from adding these baffles.
Of course I would certainly verify that conversion stays the same
by running a few more paired tests and other raw materials,
just to be sure you are not making a costly mistake.
I just want to mention to mention some important checks you should ensure about your data:
Firstly, the assumption of a t-distribution for the z-value implies the "n" pairs
of data are normally distributed.
That's an easy feature to check using a q-q plot.
Secondly, the assumptions also require that the pairs are independent.
This is often the case, but is something to be aware of.
So normally distributed values for w, and independent values
of w. These two requirements are no different to the requirements we saw earlier
to construct a confidence interval.
The last requirement is that you have to actively chose to make a paired test.
You cannot go after the fact and take a regular set of data where you compared A against B
and decide to analyze it in the way shown in this video.
You will know when you run a paired test, when there is variation
in common that you want to cancel out.
So you will then analyze it in the way we saw here.
I mention this, because it is very tempting to run all your A samples first,
then all your B samples afterwards.
I showed earlier this is the worst way to run your experiments.
Think of the case where it might take several hours to add the baffles
to the tank and then remove them again.
Your boss, and the operators, are going to really dislike adding, then removing it,
and taking this time between each test.
Will you be able to justify to them why you are making them do all this extra work?