It's a pleasure to be back here.
Thank you very much for the invitation - It's always great to come back and see familiar
old faces.
And this project that we'll be speaking about is R Package that I wrote, I've been the primary
developer on, but we started working on this together with Martin and Tommy and several
of the interns I think a couple of summers ago here, in this very room, that's right.
So here's where it all began.
This is the R Package Ranking Project, which is used to make simple visualizations for
comparing populations.
So I'll talk about exactly what that means.
Today's talk is the Demographic edition; tomorrow we'll have an Economic edition, which is essentially
the same talk, but with different data sets so that we can popularize all the good work
being done at the Bureau.
So I'll begin with Motivation.
What is the problem that we're trying to solve?
Why are we interested in this?
I'll give you examples of the kind of visualizations we've summarized and collected and developed
and then I'll talk about actually using the R Package itself.
The variable we'll be using just for the start of the talk, "Mean travel time to work" that
is from the ACS the "Mean travel time to work of workers 16 years and over who did not work
at home" in minutes.
I don't want to repeat that entire phrase every time so I'm just going to say "Travel
time to work".
The Motivating problem here is that the Census Bureau puts out Ranking Tables from the ACS
and from some other data products that we create where for instance we have for each
state in the United States.
The estimated mean travel time to work and its margin of error and so from these ranking
tables it's natural to rank them in order, see which state, for example, has the highest
and the lowest travel times to work on average and use the margins of error to compute marginal
confidence intervals so we might have 90% confidence intervals for each state's estimate.
And then an natural thing to do, if you actually plot these intervals, to say well some intervals
do not overlap, so they must be significantly different estimates and some intervals do
overlap so those estimates must not be significantly different.
That's sort of the natural instinct by following the eye, but it's not exactly statistically
valid.
Having overlapping confidence intervals for two estimates is not always equivalent to
those estimates being not significantly different; right so just because they overlap doesn't
mean the estimates aren't significantly different.
If they don't overlap, that means they are significantly different unless you have many
estimates that you're comparing and you want to allow for multiple comparisons.
In that case, then it's no longer the case that non-overlap means they are different.
So neither overlap nor non-overlap exactly translates to significance of the difference
between any pair of estimates.
And so for this reason we've been thinking, with Tommy and Martin and others, about ways
to present data in a visual format that directly shows the significance of those differences
as well as the precision of those differences in terms of a confidence interval for those
differences and other flavors of this sort.
For instance, here's a ranking table for the mean travel time to work from the 2011 ACS,
the one year estimates there, and here on the left hand side you can see the entire
table which is all 50 states plus D.C., I think
Puerto Rico is there as well.
And that's too small to read on this screen so
I blew up just a portion of it.
So we have, for each state, the geographical area, how many minutes the estimate essentially
the margin of error, plus or minus some amounts, and then on these ranking tables you can also
select the state, so here I've clicked on Louisiana so that has the two hashtag symbols
next to it to say that's the one we're using as a baseline to make comparisons from.
And then the rest of the column shows our other states statistically significantly different
from the one we selected.
So we can see that Arizona, Colorado, Tennessee, and Nevada in this case are not statistically
significantly different estimates from Louisiana's estimate.
So either they have essentially the same mean travel time to work or if it's not the same,
we haven't estimated it precisely enough to be certain which one is larger, which one's
smaller.
So this is something that is already available on the Census Bureau website and we wanted
to come up with ways to visualize this more directly than just in a table.
Also I believe these calculations of significant differences do not account for multiple comparisons
at all.
So that's not an option that you can currently select.
re only going to pick a specific pair in advance or
is it possible that we'll be scanning the whole table and looking for significance in
which case multiple comparisons,
corrections probably should be warranted.
Because all 50 states won't show up on the slide so well, I've subset just to the states
in the southern region of the United States.
So here are the Census regions and divisions.
The example for the South seemed to work pretty well for this example.
Later we'll be talking about states in the Northeast and so on.
But I think the first few states we'll be comparing are Louisiana - let's say that we're
interested in Louisiana, maybe we’re a reporter from that state and we want to compare it
to its neighbors and we have neighbors such as Texas on one side and Mississippi to the
other.
Are the estimates significantly different or not? Can we really rank these three states,
or is it going to be difficult to rank them because of the inadequate precision for that
particular purpose?
Here are the marginal confidence intervals, 90% confidence intervals for all, I believe,
17 states in the southern region from this particular data set.
So from Oklahoma has the lowest estimated value of mean travel time to work, Maryland
has the highest, and Louisiana is here in the
middle ranked number nine or so.
And again when we compare these confidence intervals you might say, “Okay well let's
see, here's Louisiana and its neighbor Texas; those intervals overlap.” So it's natural
to assume that we've measured - we haven't measured their difference precisely enough
to say that there's a significant difference between them.
But that's not necessarily, actually true.
And same for Louisiana and Mississippi; they're very close but they do not quite overlap.
Here we see Mississippi is - it's confidence intervals upper end is just to the left of
the lower end for Louisiana's confidence interval so they do not overlap; whereas Louisiana
and Texas do have intervals that do overlap.
So the question is can we really rank all three of these states?
Do we know that this is the correct order or is it possible that these should be flagged
as not being significantly different and given the same rank?
Here's a subset of the table.
We have the three states, the estimated mean travel time to work in minutes, the estimated
standard error from the American Community Survey, and I'll walk briefly through how
we
could check for the significance of each of those differences.
If we were only looking at one difference at a time and not thinking about multiple
comparisons, one way to do that is build a confidence interval for the difference between
the two states.
So instead of looking at the two separate confidence intervals for Mississippi and Louisiana,
we build one confidence interval for the difference between them.
So we take the difference between Mississippi's 23.86 estimate and Louisiana's 24.54 estimate,
and then we do plus or minus the z-score; if we want a 90% confidence interval the appropriate
z-score is 1.64, and then we get the standard error for the difference between them, which
we calculate by taking the square of each standard error summing those taking the square
root of that sum.
So this is standard, straightforward calculation for a confidence interval of the difference
between two independent estimates.
That's something I should have also prefaced this with, is that because of the way the
sampling
is done, we can assume that the estimates themselves
are independents from state to state - in
the American Community Survey at least.
And so once you crunch all these numbers you see that the confidence interval for Mississippi
minus Louisiana is between -1.1 and -0.2.
It doesn't include zero.
So that means the possibility that there's no difference or that the difference could
be either positive or negative since it's not in that confidence interval for the difference,
we can say okay we feel confident that we did measure this correctly, with adequate
precision, this really is the right ordering.
Mississippi's estimate is lower than Louisiana's and we're confident that Mississippi is indeed
- has a true value of the mean travel time that's lower than Louisiana's.
Now remember for Texas, we saw that the marginal intervals overlap with each other and so we
might think that they're not going to be significantly different but if you do the same calculation
for Texas, you see that it just barely excludes zero as well - on the other side, so Texas
minus Louisiana, that confidence interval turns
out to be 0.01 to 0.55.
Since it also doesn't include zero, the difference between those two states' estimates is also
statistically significant.
So this is essentially just illustrating the point that just because two intervals overlap
- the marginal intervals overlap, we can't necessarily say that the difference between
them is not significant.
So if we know that to be true we'd like to come up with other visualizations that might
help us make the correct inferences from the - from the plot directly.
So one way to do that would be just to plot the difference and it's confidence interval
directly.
So here is the - Louisiana minus itself at a
baseline of zero that's going to be estimated
as zero always.
Then Mississippi minus Louisiana is going to be the estimate and the confidence interval
for that difference that we came up with on the previous slide.
And a similar thing for Texas, and we can see by looking at these plots directly that
because the confidence interval for each difference doesn't overlap zero then that translates
directly to saying these differences are statistically significant.
That is if we're assuming only one comparison is being met at a time on the other hand if
we want to do multiple-comparisons-corrections, we might say, “Well we have the seventeen
states in the South,” that we showed earlier in the talk, “so let's do a Bonferroni correction
for the fact that we could compare Louisiana to 16 other southern states”.
If we show all 17 of those estimates at once and Louisiana is their baseline, that's 16
possible comparisons.
So we incorporate that into the z-score, inflating that to 2.7, instead of, it was something
like 1.6 before.
And now the confidence interval for the difference - Mississippi minus Louisiana - that interval
does include zero.
It goes from -1.5 to 0.1 essentially.
So because that difference has an interval that does include zero, we can no longer say
that they're statistically significantly different after we try to account for the fact that
there are multiple comparisons possibly being made.
Essentially, we could do this for each of the states.
We could compute these intervals for the difference between Louisiana and each of the other southern
states, we could allow for a Bonferroni correction for the number of comparisons being made.
And now we have, as before, Louisiana has a difference with itself is always going to
be estimated at zero, but the other states are presented all on one graph.
And now it's legitimate to look at any one of these intervals, see whether it overlaps
the line at zero through Louisiana, and say for any one of these: Is it significantly
different from Louisiana or not?
The multiple comparisons have been accounted for, the natural instinct of looking at overlap
of something with something else, is dealt with in the - in the method of looking at
whether each state overlaps with Louisiana in particular.