Important statistical concepts: Significance and usefulness, evidence and
strength, Association and causation.
Significance. When a result is statistically significant,
this means that there is evidence that the effect that we have observed in the sample
exists in the population.
The p-value tells us if a result is significant or not.
A p-value lower than the significance level
tells us that the result is significant.
It is unlikely to have occurred as a result of sampling error.
A large effect will usually be significant.
Choconuttie example. The null hypothesis is that the mean weight of
peanuts in the 200 gram choconuttie bar is 70 grams or more.
The alternative hypothesis is that the mean weight is less than 70 grams.
we decide on an alpha value of 0.05.
We take a sample and the mean weight of the sample is 50 grams.
This is a big effect as 50 grams is considerably less than 70 grams.
Even with a small sample of 10 bars we are unlikely to get this result
if the null hypothesis is true.
The p-value will be small and the result
will be significant pretty much no matter what size the sample is.
In contrast a small effect may be significant if the sample is large enough.
Say the mean weight of the sample was 68 grams of nuts
Is this statistically significantly less than the hypothesized 70 grams?
Is this 2 gram difference enough that we can reject the null hypothesis?
Well it depends on the size of the sample.
If we had a really big sample a difference this small might be statistically significant
With a small sample it will not be.
The sample size is a factor when calculating the p-value
"Statistically significant" does not mean large.
Statistical significance should not be confused with practical significance or usefulness
If the mean weight of the nuts in the bars in the sample is 68g,
with a very large sample this will show up as statistically significantly lower than 70 grams
But it's probably only 2 grams lower
a difference of 1% of the total weight of the bar.
No one is going to notice.
Though the result is statistically significant it is not important in practical terms.
We need to be careful because the everyday meaning of significant is "important" or "notable".
When scientists report something as significant they
mean there is evidence of a result in the population.
When journalists report something as significant they mean big and important.
Evidence and Strength
This is pretty much the same idea but related to regression and correlation.
It is tempting when the p-value associated with the slope is low
to state that there is a strong relationship between the variables.
However this is not correct.
A low p-value for regression tells us that there is evidence that the slope is not zero.
A low p-value does not tell us that there is a strong relationship between the two variables.
To examine the strength of the relationship,
we look at the slope, the correlation coefficient and the r-squared value.
The slope tells us the change in Y relative to the change in X.
The correlation coefficient tells us how close the points are to the fitted line.
the R-squared value tells us how much of the variation in Y is explained by X.
As with all statistical tests the p-value tells us how much evidence we have to
reject the null hypothesis.
The smaller the p-value is the more evidence we have.
It is not an indication of strength or usefulness.
Association and causation
When two variables are correlated it does not mean that one causes the other.
It means that there is a linear relationship between the two variables
Similarly in regression the equation models the behaviour of the sample.
It does not imply causation.
In the videos on scatter plots and regression
Helen explored the effect of temperature on sales of choconutties
She found that for each degree of temperature
the sales of choconutties decreased by 2.4 packets on average.
We cannot say that the extra degree of temperature causes the reduction in sales.
There can be several possible explanations for a result that A and B are correlated.
It might be that A does cause B.
It could be that in this case that the extra degree of temperature causes sales to drop.
Or B might cause A
in this case it would imply that the reduced sales cause the temperature to go up.
This is unlikely in this case.
It may be that another factor is causing both A and B.
Again this is unlikely in the choconutties case.
Or there may be an intervening variable.
It could be that on hot days people feel less inclined to buy chocolate so the sales reduce
or it could be that on hot days Luke brings out his choco ice stand
which draws customers away from Helens choconutties stand
or perhaps Helen gets grumpy on hot days and scares away customers.
There are many possible scenarios.
Association is not the same as causation
It generally requires a controlled experiment to prove causation so that
other factors are taken into account.