So this is one type of visualization that we could use to address the - the issue that
just showing the marginal confidence intervals doesn't always lead to the right answer.
Again, just a recap of the Motivation as we discussed at the start of the talk, having
overlap of the marginal confidence intervals doesn't necessarily mean that things are not
significantly different; having non overlap also doesn't necessarily mean that they are
significantly different.
So we're gonna try and visualize these things directly.
e significance of differences - that's all it shows - it
doesn't have the intervals plotted on this Ranking Table in this column for statistical
significance.
So we could make a visualization that shows exactly that.
But for all the comparisons being made at once, maybe to be a little more compact.
Instead of having to click through each state one at a time, we can have one graph where
each column of the graph is equivalent to what the column in the ranking table would
look like if you click on a different state.
So let's walk through this.
On the left hand side we have essentially the ranking table that we had before; we have
the estimated rank, the name of the state, the point estimate and its standard error.
And here, let's say, we've selected Louisiana again.
So here we look for Louisiana on the bottom axis and this column is going to be white
where the difference is not statistically significantly different and then different
shades of gray for states that are significantly higher or significantly lower.
Where again, here we've done this demi-Bonferroni correction, like we talked about in the sense
that we're assuming Louisiana is the one baseline we're interested in and we're correcting for
16 possible tests being done at the same time.
If you switch your focus to a different column, you can use a different state as the baseline
and then look at the comparisons for that other state versus the other 16 states.
There's a question as to whether the demi-Bonferroni for just 16 comparisons or the full Bonferroni
for all pairwise comparisons is more appropriate for this graph, again depending on the target
audience, that's something to think about.
But the visual form wouldn't work both ways - it's just the actual way it comes out would
look a little different.
So this is one visualization.
The next one is one that we saw a few slides ago, but I'll reiterate it just to say this
is another thing that we've implemented in the Package and in the associated paper - is
these confidence intervals for differences directly.
So just as we were doing before, you can choose here Louisiana as a baseline and for each
state compute the confidence interval for the difference between - Maryland and Louisiana;
D.C. and Louisiana and so forth.
We are highlighting in bold the states whose estimate is significantly different and a
lighter line type for the ones that are not significantly different.
All right, so here's another plot, which, if the previous ones raised questions,
we'll see - we'll see what happens with this one.
So the - the concern that you might have about the one we talked about just a minute
ago, confidence intervals for the difference directly, is that it doesn't actually show
anything about the precision of Louisiana directly.
It doesn't tell us how precise Louisiana's estimate is.
So could we come up with something that shows both the precision of our reference state
- the precision of Louisiana's estimate - and the somehow relative precisions of all the
differences with Louisiana.
So this is what is called a “Comparisons Interval Plot”.
This is based on work by Almond Louis DuCanyon and we thought it was an interesting exercise
to implement this, although the interpretation can be challenging so I'm not sure that we'd
recommend this for a general audience, but I'd like to go through this and sort of discuss
how this compares to the plot we were just looking at.
So the gray shaded area in the middle, you can see on the bottom x-axis, is exactly the
confidence interval for Louisiana in this case.
So for Louisiana its confidence interval goes basically from 24 to 25 minutes mean travel
time to work and that's what's shown here.
For each other state an interval is constructed such that if that interval overlaps with Louisiana's
interval, then that pair is not significantly different.
If this interval doesn't overlap with Louisiana's grey shaded interval, then they are significantly
different.
Again, I know that wording we might tighten up a little bit.
But these are not usual confidence intervals for the other states.
So here on the top x-axis we show - with the same units as below, but just with a different
baseline - we show is zero at the edges from Louisiana.
And you can look at the distance between Louisiana and the edge of one of these other intervals.
So that distance itself - so not the actual interval being plotted - but the negative
space
between this strip and this other interval, that is the confidence interval for the difference
between the state and Louisiana.
Let me show a schematic illustration that might help.
So here, let's say we have Louisiana as a reference state, here's its point estimate
and here is its regular marginal confidence interval just for Louisiana itself.
Here's another state with an estimate that is lower than Louisiana's estimate.
We'd like to construct an interval with the following property.
The distance between this interval’s upper bound and Louisiana's lower bound is one end
of the confidence interval for the difference between these estimates.
The difference between the outer edges between this guy's lower bound and this guy's upper
bound, that distance is the other end of the confidence interval for the difference between
them.
It's a way of summarizing a lot of information on one graph, because using the upper axis
and the lower axis together you're able to read off the same information that we had
on the earlier plot with the bonus of having the information about Louisiana's precision
itself.
So there are arguments to be made for this presents a richer set of information than
the previous plot did, on the other hand you do have to work a little harder to get that
information out of it.
It's - it's a little counterintuitive to have the sort of the negative space between intervals
as the thing of interest, rather than the intervals themselves.
So for that reason it's worth taking caution with this visualization, but it is one that
we've seen in the literature that we thought was an interesting concept that's worth exploring
and we chose to implement it in the Package as well.
So there's one more slightly mathy concept to go over before we get to the final
set of plots.
This is an idea from a paper by Goldstein and Healy and so we're calling these Goldstein-Healy
Corrections, in a sense.
So say that we have again, two populations.
Let's talk about concretely Louisiana and Texas and we'd like the intervals to have
the property that the actual intervals for each of those states - the Louisiana interval
and the Texas interval overlap if, and only if, those two estimates are significantly
different from each other.
One way to do that is by changing the confidence level of the marginal intervals that we're
plotting.
So instead of showing 90% intervals for each of the two states and then being disappointed
that their overlap doesn't really correspond to significance of the difference, we could
find what's the right confidence level to show.
So in this case it I think it's something like 80%; if we happen to show 80% confidence
intervals, then the overlap, or not, between those two intervals actually does correspond
to a level 0.1 test of the difference between them.
So this is a calculation that I haven't gone through here but it's straightforward to do
- both the American Statistician paper that we've submitted and the research
report; I think we have a 2014 research report here that goes through the calculations in
more detail.
But, essentially you can choose if you'd like the test of comparing any two
populations to have Level Alpha, so for our case we might want Level Alpha equals
,700 0.1.
We can choose a Alpha GH - Alpha Goldstein-Healy - that is a different Alpha form actually
showing the confidence intervals.
And when we show those confidence intervals with - with our correctly chosen Alpha, then
whether or not the overlap is actually equivalent to this test.
So that works for having just two populations.
If we have more than two populations, you're not guaranteed for this to work across
all pairs, but something that you can do to approximate that is to say that, okay on average,
across all of the K, choose two pairs that we're comparing the level of all those tests
on average is the target level of Alpha.
I know that's not entirely satisfying - it's not a guarantee to be conservative at all,
but it's an alternate approach to showing everything on one slide, on one graph,
with approximately the right coverage or approximately the right level for each of those comparisons.
With all that said about the Goldstein-Healy idea, when you plot it, it just
looks like a usual set of confidence intervals.
It looks like a bunch of a marginal confidence intervals, but now they've been adjusted to
be a little shorter - each of them has a slightly lower confidence level then the usual marginal
ones would, so that now you can take a particular pair, compare them for overlap and say, well
it has this average significance level - at this average significance level of alpha,
it is or it is not significantly different.
And then what we realized is you can add more information to this by making what we're calling
two-tiered error bars or two-tiered confidence intervals, where the inner tier is these Goldstein-Healy
adjusted versions.
So you can compare overlap of the Goldstein-Healey ones by looking at the inner intervals between
the vertical spikes - vertical lines, but the
outer lines are the usual 90% confidence intervals.
So you get both sets of information. For each state you see, it's 90% confidence interval,
you see the precision of that state directly.
But you also have these Goldstein-Healy corrected versions which you can - whose overlap you
can compare in a legitimate way, if you accept that this averaged confidence level is - average
significance level is something that's appropriate; or as per this suggestion, if you do the
conservative version, then you get conservative but correct testing as well.
And finally, this has been so far with no Bonferroni correction, let's say that we want
to do a demi-Bonferroni correction again, you might want to do a full one instead, but
here let's - let's show the demi-Bonferroni correction.
We're showing each individual confidence interval as before, the individual 90% intervals.
Now we're doing Goldstein-Healey and a demi-Bonferroni correction at the same time and what that
did was made the corrected intervals become wider than the individual ones.
So now the rule is of which is the inner tier and which is the outer tier have flipped.
So we want to label that directly on the plot to make it clear.