My name is Yea-Seul, and I am an assistant professor at UW Madison.
This work is in collaboration with Jake Hofman and Dan Goldstein in Microsoft research.
As the world becomes more data-driven, people are increasingly exposed to statistical information
about uncertain outcomes.
But how well do readers understand the effect size?
What is effect size?
Imagine two different experiments showing that the treatment group who drank green tea
had reduced bad cholesterol by 2.2 mg on average.
One experiment shows that the outcome of the treatment group varies a lot around the group
mean and considerably overlapped with the control group who didn't receive any treatments.
In another experiment, the outcome of the treatment group seems less varied and less
overlap with the control group.
Statisticians call this measure of outcome variability of the treatment as effect size.
While understanding effect size can impact your decision, communicating this measure
is not a simple task, and even worse, most of the time, articles omit any information
regarding the outcome variability.
So in this work, we explore several research questions.
First, How effective do people think a treatment is when the treatment is summarized only in
terms of its average effect And second, How do these initial perceptions
change after people are presented with information about effect size?
And in the third study, we try to see if showing more information can make people's perceptions
more accurate.
And in the last study, we wanted to check how robust our findings are.
I am not going to present studies 3 and 4, but please refer to our paper if you are interested.
To address these research questions, we present our study with an imaginary scenario of playing
a game using a boulder.
© participants were told that they are athletes competing against an equally-skilled opponent
named Blorg.
To win this game, you need to slide your boulder farter than the opponent.
And you have a chance to rent a premium boulder that slid farther than the standard boulder
that Blorg will use.
We then ask how much you are willing to pay for renting and the estimates of the probability
of winning if you rent.
Both questions approximate the perceived effectiveness of the premium boulder, the effect size of
the premium boulder, and the premium boulder is a treatment in this case.
In our scenario, we set the effect size of the premium boulder to be small since this
is the most prevalent case in news articles and scientific papers.
In the first condition, we simply show a directional statement illustrating the fact that the premium
boulder slid farther than the standard boulder that Blorg will use on average.
Then we asked two different questions: we asked how likely it is for you to win with
premium boulder when Blorg uses a standard boulder.
Also, we asked how much you would be willing to pay to use the premium boulders.
We have four more conditions that explain the performance of the standard and the premium
boulder.
In the absolute difference condition, participants saw the statement "The premium boulder slide
4 meters farther."
In the percentage difference condition, we showed "The premium boulder slide 4% farther."
In the statistical significance condition, we described the 95% confidence interval and
the average sliding distance of two types of boulders.
In the last condition, we showed the visualized confidence interval alongside the textual
information in the statistical significance condition.
We ran this study on Amazon Mechanical Turk with a total of 750 participants.
This visualization shows the result of the chance of winning.
The X-axis represents conditions, and the y-axis represents error.
Being close to zero means that participants in that specific conditions perceive closer
to the ground truth.
When we illustrate the performance difference in percentage difference, participants' perceptions
were closest to the ground truth answer.
However, participants in a condition where the 95% confidence interval presented with
the visualization overestimated the true effect size more than any other condition.
It was alarming since they are the most commonly used format in scientific papers.
We observed a similar pattern in error in willingness to pay.
In study 2, we investigate whether we can fix this overestimation problem by showing
information about outcome variability.
In study 2, first, we show statistical significance with the confidence interval visualization,
which led to the most error in perception in study 1.
And this was the same for all the conditions.
Again ask the same questions about the chance of winning and willingness to pay to infer
participants' perceived effect size.
And then, on the next page, we vary to show information that directly and indirectly illustrates
some form of outcome variability.
In the categorial condition, we described how small the difference was between the standard
boulder and the premium boulder.
We aimed to replicate how Cohen's d effect size is often communicated using a term like
small, medium, and large effect size.
We compare their initial response to the revised response after seeing the outcome variability
stimuli to see if these interventions help alleviate the overestimation problem.
In variance condition, we stated the variance that participants could sense how the outcome
from two boulders can be varied.
In the probability of superiority condition, we said how likely the premium boulder would
win if the game is played 100 times.
As you noticed, this condition is directly giving the participants an answer to the chance
of winning, therefore only use the willingness to pay as a measure in study 2.
Next, we provided a height analogy that participants might be familiar with.
We are inspired by Cohen, who introduced the measure of standardized effect size.
He actually used the height analogy in his book, explaining how small effect size can
be translated into the height difference between 15- and 16-year-old girls.
We formulated one more analogy condition that used the weather in New York City as well.
We ran this on AMT with 1,800 participants.
In this study, we only compare the willingness to pay since the chance of winning was one
of our stimuli to communicate effect size.
The length of the arrows indicated how big adjustment participants made from their original
misperception measured by willingness to pay.
The dotted line represents the ground truth.
Probability of superiority, which essentially describes the chance of winning, height analogy,
and variance reliably reduced error in willingness to pay.
Study 3 looks into whether the pairs of effect size information can further alleviate misperception,
and study 4 demonstrates the robustness of our findings.
You can refer to our paper for further details.
In this work, through multiple studies, we demonstrated that:
common ways of summarizing mean difference, such as absolute difference or 95% confidence
interval with and without visualization, can cause overestimation of treatment effectiveness
Showing probability superiority can be effective to alleviate the overestimation.
More importantly, the analogy can be about good as stating the probability numerically.
If we translate our study results into the probability of superiority, there was a 62%
chance that people who received outcome variability information made better inference than those
who did not And this is about equal to the probability,
among American women, that a randomly selected 18-year-old is taller than a randomly selected
13-year-old.