Let's look at the histograms of f0 and f1, and f0 and f2 again.
Let's switch to the iPython notebook quickly.
Here we are looking at two histograms of f0 and f2.
The f0 is the average submitted charge amount shown in green,
and f2 is the average Medicare allowed amount shown in blue.
Notice the shape of the two histograms are quite different.
Moreover, the value ranges for f0 and f2 are also quite different.
Similarly, you can look at the histogram of f0 which is, again, in green.
And f1, which is the Medicare payment amounts, given in red.
Here also you see a difference in shape, and range of the histograms.
To compare two histograms in the same range, we can scale the variables.
To scale variables, we can subtract the minimum value of
the variable from each of those valleys of the variables.
And divide by the range of the variable which is
the maximum value minus the minimum value.
Let's now use this formula to scale each of our variables, and
re-plot our histograms.
Upon running, you will get something like this.
It is very hard to see here.
So in the next histogram we will make this into a logarithmic scale.
To change the y scale to logarithmic, we have changed log equals 1 in the code.
Let's run this part of the code.
You should now have a histogram that looks like this.
This is the histogram of f0 and f2 plotted in a logarithmic scale.
Also notice the range and the x, or the horizontal axis.
The range of the scaled variables are between 0 and 1.
Of course be careful, such that you don't have only one data point,
in which case you will get 0 for the denominator because your max and
min values will be equal.
Scaling the variables makes them computationally easier to deal with,
and easier to visualize.
Now we are going to look at correlations between two variables, but before we do
so, let's look at where we are in our questioning and model-building phase.
So far, we have inspected the various variables of interest in the data.
We have looked at the allowed amount, charged amount and the payment amounts.
Now we will ask the correlations.
In asking if the variables are correlated,
we can ask if the correlation in non-linear, or if the correlation is linear.
We can inspect this by looking at a scatter diagram of the variables.
Let us now inspect the scattered diagram of the two variables f1 and f2.
Notice that we have plotted both the variables after scaling them.
Once you are done, you should have a graph that looks very much like this.
The scatter plot of these two variables are very linear.
What we can conclude from that,
is that the paid amounts are directly proportional to the allowed amounts.
Similarly we can look at the scatter plot of f0 and f1.
In this case, although we have some correlation,
it is not as linear as it is between f1 and f2.
We want to have a quantitative measurement of the correlation between
two variables.
We have already observed from the scatter plot of f1 and f2,
that they're linearly correlated.
We can calculate what is called the Pearson's Correlation Coefficient.
Let's first define this quantity.