In this lecture, we're going to look at the term "disturbances".
We're going to gain an insight into why we randomize our experiments,
the concept that we've seen several times in the course so far.
In many ways, this is one of the more important videos.
Now, I could have covered the topic of randomization a little earlier,
but I needed the terminology of confounding from the previous class
to clearly explain the purpose of randomization.
Also, by waiting, you've started asking really good questions
about randomization on the course forums.
One of the best times to learn anything is just when you need
that information, just in time learning.
Now that you've got all these questions in your mind
about randomization, we're ready to learn about it.
To do so, we must understand the nature of variables or factors in our experiments.
We can categorize our variables in several ways.
The first way, is to talk about variables that we know about,
and those that we don't know about, the unknowns.
The second way, is we can talk about variables that we can control
and those that we cannot control.
And thirdly, we can consider variables that we can measure and those that we cannot measure.
We mostly deal with variables we know about.
So the first distinction is a little unnecessary,
but we will see where we use it later on.
There almost certainly are variables in your system that will affect your outcome,
that you did not think about when you start experimenting.
And you'll see why these unknown variables can play an important role later on.
I 'm going to walk through an example to show what I mean
by controllable and measurable factors.
Once you understand the terminology, you will see why randomization is so crucial.
A great example was posted on the forums about why randomization must be done.
Lets take a look.
Imagine we're baking ginger biscuits and I'm investigating
that with eight experiments because I have three factors.
I decide I'm going to do all these experiments in one day.
So I start in the morning and by the end of the day, I'm going to be pretty tired.
When I mix the ingredients by hand, I may not be doing it so well by the end of the day.
Tiredness is something I cannot control in the experiments if I do them all in one go.
And I am also not able to measure my tiredness.
But being tired will affect some of the outcome variables.
Such as the texture of the biscuit if I am not mixing things properly.
A disturbance is defined as something that you do not have control over,
and something that you are not able to measure.
So tiredness is a variable in our system that we call a disturbance,
it's uncontrolled and unmeasured.
I'd like to note here that the term control refers to your ability to adjust the variable.
When we say you cannot control something, it means we cannot actively change it.
For example, you are not able to change the temperature outside or the humidity outside.
But you can control the temperature inside your house if you have air conditioning
and you might also be able to control the humidity in your home.
When I say that you're not able to measure something,
I don't mean that it's impossible to measure it or quantify it.
I just mean that you might not have the right tool, or the instrument, or way to do it.
Humidity could affect my system, but I might not have enough money
to buy a decent humidity sensor.
That's why tiredness is a disturbance.
I cannot control it, and I don't have a way to reliably measure it.
Back to the baking.
You can visualize that there's this increasing tiredness factor taking place over time.
Now here are the eight experiments written in standard order.
If I chose to run the experiments in that same order,
you will notice the third factor, the effect of baking time.
We'll have four low level experiments first and then four high experiments last.
Those first four experiments were when I had lots of energy and mixing my ingredients well.
The last four experiments were run when I was tired.
What has happened here is that we've confounded the effects of tiredness
with this last factor of baking time.
If I analyze my experimental results,
I might find that baking time has an important effect on the outcome.
Was it baking time that caused it or was it me getting more
and more tired throughout the experiments?
You can also imagine a situation where we find that baking time has no effect.
But maybe baking time actually did have an effect, but it was counteracted or cancelled
out by this unmeasured tiredness factor due to confounding.
Both hypothetical examples there would lead to a wrong conclusion.
Now recall in an earlier video, I had said
that computer simulation experiments generally don't need to be randomized.
Now you can see why.
Those types of experiments generally don't have unmeasured
and uncontrolled disturbances happening.
If you repeat the computer experiment, today, tomorrow or next week,
we should be obtaining the same results.
So you must randomize your experiments if you are not running your experiments
in a very controlled environment like a computer simulation.
In the discussion forums, the number of you have proposed experimental designs
that might be confounded by disturbances, if you don't randomize the trials.
For example, if you experiment with a way to learn a new language or multiplication tables
for your child or going to gym or memorizing 20 digits, every time you do one
of those experiments you are naturally going to get better and better.
Simply because you are practicing and not necessarily due to the experimental conditions.
In other systems, simply experimenting makes the system worse and worse.
For example, if you run tests for gas mileage,
your car is naturally deteriorating over time at a very slow rate.
In the baking experiments, I was getting tired.
People in the chemical industry know that our equipment slowly deteriorates over time,
and we shut it down periodically to clean it out and restart them.
You can thank the second law of thermodynamics for that unhelpful feature.
Now you can see why it is so important to randomize.
It ensures that our system is minimally affected by variables which we cannot control.
Minimally affected by variables which we cannot measure and also minimally affected
by variables which we don't even know about.
When we randomize, we are doing so, so that we're not confounded or confused
by the uncontrolled and unmeasured disturbances taking place.
However, there are disturbances that we can measure.
The advice here is that you should record such variables and add them
to your table of results as extra columns.
In the baking example, I may not be able to control the temperature
in my house, but I can still measure it.
It might be possible that I could measure the humidity.
Humidity plays an important role for certain baked products and if I can measure it,
I should do so and add it to my table.
We call such additional measurements covariates, which is a term for a variable
that is not really our primary focus, but might still vary
and potentially influence the outcome variable directly.
Or, as is more likely the case, that covariate will affect one of the factors
in our experiment, which then in turn influences the outcome.
You can use this extra information from the covariates in two ways.
Firstly, the most simple way is to simply use them
to understand unusual results after the fact.
If one of your experiments had, for example, an unusually low outcome value,
the reason might be due to a covariate.
The second way we could use it, for those of you that have an understanding of Least squares,
is we can add these covariates as additional regression variables
in our model to separate out their effect.
There's more to say on that topic, but now that you know that terminology, you're in a position
to do the extra research outside of this class and see how you could deal with covariate data.
Let's test your knowledge.
We're going to use this example in the next video, so try to remember it for next time.
The example was inspired by an article that appeared in the Harvard Business Review.
You are marketing a calendaring app for a cellphone, lets call it Cal app.
The basic functionality in the app is free,
but inside the app the users can pay small amounts to upgrade various features.
These are known as in app purchases.
For example, you could pay $1 and get the sync-to-other-devices feature
or pay another dollar extra for SMS reminders.
Or you might really need that extra feature for integration
with your desktop calendaring application, and you guessed it,
that's going to cost you another dollar.
So your company has created the app, but it's your job to sell and promote it.
Each experiment you do might involve about 2,000 people, and you'll measure the percentage
of those people that are still using the app after 60 days.
That's your outcome variable.
The factors that you consider might be, for example,
factor A is whether you provide a single free in app upgrade for one of the features
or at the plus level you provide a 30-day trial for all the features.
Factor B might be that the sales message is: "This app has your schedule available
at your fingertips on any device".
Or the alternative message: "Cal app features are configurable,
only pay for the features you want".
Factor C might be, the price for in app purchases is $0.89 per feature.
Or at the plus level, the price might be $0.99 cents per feature.
So those were the factors.
Now consider these other variables.
Determine whether they are disturbances or covariates.
Note that a variety of answers could be correct, depending on the assumptions you make
and on your knowledge regarding the capability of cellphone apps.
So, in summary, the distinction between covariates and disturbances is
that covariates are measurable, while disturbances are not.
Neither of them are controllable.
In the next video, we are going to consider a subtle difference
in the variables that can be controlled.