The blind audition study is one of the most celebrated social science papers
of all time. It's famous for showing that when orchestras auditioned musicians
blindly behind a screen, women's success rates soared. But it turns out, the study showed
no such thing. What's going on? That's coming up next on the Factual Feminst.
During the 1970s and '80s, the nation's
orchestras became more open and democratic, and to ensure impartiality,
several introduced blind auditions. Two economists, Claudia Goldin of Harvard and
Cecilia Rouse of Princeton, noticed that women's success rates increased along
with the adoption of screens. Was that a coincidence, or the result of the screens?
That's the question they tried to answer in their 2000 paper
"Orchestrating Impartiality: The Impact of Blind Auditions on Female Musicians."
They collected four decades of audition records and rosters from eight major
orchestras and crunched the numbers. Now, their paper includes multiple warnings
about small sample sizes and contradictory results — but few readers
seem to have noticed. What caught everyone's attention were some strong
claims in the final paragraph: "We find the screen increases by 50% the
probability that a woman will be advanced from certain preliminary rounds."
According to Google, the study has received 1,500 citations in academic
articles and thousands of media mentions, and showcased in so many diversity
workshops that one recent attendee begged never to hear about it again.
Now, the study's appeal is clear: two prominent economists, in a top journal,
wielding state-of-the-art econometrics, captured and quantified gender bias — and they
also documented a solution. Or so it seemed. The research went uncriticized
for nearly two decades. That changed recently when a few scholars and data scientists went
back and read the whole study. They found a tangle of small, ambiguous,
contradictory findings. For example, the screen seemed to help women in
preliminary audition rounds, but in the semi-final rounds, they didn't.
And none of the findings were strong enough to draw a broad conclusions one
way or another. And the authors say as much, albeit ambiguously throughout the paper.
So where did Goldin and Rouse get their
totemic conclusion that blind auditions dramatically improved the success of
women candidates? Well, after warning that their findings
were not statistically significant, they simply declared them to be
economically significant. But what does that mean in this context? "That doesn't
mean anything at all," says Columbia University data scientist Andrew Gelman.
In a recent commentary on the study, he said that they're just "fine words" that
really mean that our data is "too noisy to form any strong conclusions." Now my guess
is that the authors thought they detected something with real-world
relevance somewhere in all that noise — but that's a reason to call for
more research, not to declare the transformative power of screens in
women's quest for equality. Still some may think it seems obvious that the
screens contributed to equal hiring, but it's not. The screens may have been a
reflection of changing attitudes — and it was those attitudes, not the screens, that
helped women. After all, women didn't need blind auditions to move ahead in law or
business, medicine, or even in the Cleveland Orchestra, which had not
resorted to blind auditions, according to the study.
Now, Gelman and other critics — they don't deny the reality of gender bias
and they don't question the potential merits of blind auditions
as a means of achieving impartiality. But Goldin and Rouse verified nothing about
the special benefits for women, and nor has anyone else.
The subsequent research on blind recruitment is just a morass of baseless
claims or retracted statements, contradictory findings... So how did such
an equivocal study achieve iconic status? Well, a lot of the credit goes to
confused but influential fans, such as the writer Malcom Gladwell and the
Harvard psychologist Mahzarin Banaji. They saw the study as powerful,
indisputable proof of the ubiquity of gender bias, as well as a way to counter it.
According to Gladwell, "Orchestras in the 1980s started putting up screens in
audition rooms and immediately — immediately — orchestras
started hiring women left and right." And here's professor Banaji in a
2017 TEDx talk: "In the late 1970s, American orchestras were almost entirely
all male . . . Once the curtain dropped, the case study shows that the number of
women who were selected doubled — they went up 50%." But it didn't show that.
Truth matters. Overhyped claims create confusion and undermine public trust,
and they don't solve problems. Sex discrimination in the workplace is a
serious matter, but improvements require solid data,
replicable research, and careful evaluations of causation. As the scholar
Alice Dreger says, "Carpe datum . . . Evidence is an ethical issue."
If you found value in this video, please show your support by subscribing to the series,
and follow me on Twitter, and listen to my podcast, "The Femsplainers."
Thank you for watching the Factual Feminist.
Wave goodbye, Izzie.