My husband John and I are philanthropists.
That word means different things to different people.
To us, it means aggressive investments in a number of issue areas:
from evidence-based policy to criminal justice, education,
health care, pension reform, public accountability, and many more.
I get asked a lot of questions about our work:
why did you choose the issue areas that you did
and how do you choose new ones?
I always feel that when responding to that question
people are expecting me to tell a story about some transformative event
that happened in my life or John's life that led us to embark on this path.
Maybe they want to hear that we invest in healthcare
because we've had family members who unfortunately have had cancer.
Maybe they want to hear a story about my life growing up in Puerto Rico
and how that fueled
my interest in and passion for fighting social justice issues
and addressing poverty.
At TED, we've come to expect that every talk will start
with a story like that: something intensely moving and personal.
Maybe something that starts with: "When I was a kid."
But I'm not going to tell a story like that, although I could.
I think personal stories are so important; they are what fuels our passion.
For many of us, they're the reason why we do the work we do.
But I believe that they're precisely the wrong place to look for insight.
In fact, I believe that we -
as policymakers, government officials, philanthropists, even as individuals -
spend entirely too much time on anecdote and not enough time on evidence.
I'm here today to argue that we need to change that.
That we're routinely making all kinds of decisions
based on incomplete, inconsistent, flawed, or even nonexistent data,
based on anecdote.
That this is harming all of us in ways that we're not appreciating.
And that we can and should do better.
Now, I've just made a bunch of provocative statements,
so let me get to work at convincing you.
I'll start with one example from the TED Global stage in 2012,
the second-most viewed TED talk of all time.
Amy Cuddy - psychologist, Harvard Business School professor -
famous in large part for her TED talk for her work on power posing.
So, she basically argued that, and continues to argue,
that standing in a power pose position
like WonderWoman, like this or like this, whatever makes you feel super powerful,
changes the hormone levels in your body
and might even be conducive to greater success.
It's amazing, right? I mean, that's so cool.
I should just stand like this the entire time.
The TED community ate this up:
over 38 million views over the last four and a half years.
The talk has been subtitled into 48 languages and counting.
Here's how the TED website today describes Professor Cuddy's work:
It says that Professor Cuddy's work reveals the benefit of power posing,
and it makes a reference
to this hormone level issue I just talked about.
National news outlets gushed about the work,
and many of us rushed to buy her best-selling book.
Now, I hate to be a downer,
but I think we were all duped.
Power posing doesn't work - or at least not as advertised.
Look at the myriad studies
that have been conducted on power posing
ever since Professor Cuddy released her findings.
They point to glaring errors in her methodology;
they question the entire theory of power posing.
Some researchers tried to reproduce her work and couldn't;
others tried to reproduce her work and got exactly the opposite result:
that power posing reduces feelings of power.
And it gets worse.
Even Professor Cuddy's co-author, who herself is a prominent academic,
has completely disavowed the study, and she did so in no uncertain terms.
Here's what she said: "I do not believe that power pose effects are real."
(Laughter)
And she goes farther than that.
She confesses to glaring errors in the way that they conducted their study:
tiny sample sizes, flimsy data, selectively reported findings.
Even Professor Cuddy herself,
who continues to stand by the power pose theory,
now has sort of amended her story,
and now she says that she's agnostic
about power poses' effects on hormone levels.
Agnostic?
That was the whole reason she went on the TED stage and did this and did this
and made us all do this and feel powerful.
You might think this is a trite example,
and it's by no means meant to be a personal attack.
Who cares if I stand in front of you like this,
if that makes me feel good or not?
And what does that have to do with solving the country's problems
and the world's problems, which is why we're all here?
Well, a lot.
Because this example is endemic
of something that we see throughout academic research
and throughout research in general.
Virtually everywhere you look, you'll find researchers,
many of them prominent and most of them well-intentioned,
actively misleading us into believing that bad research is proven fact.
We've seen this time and again in our work.
When John and I started the foundation,
we set out to solve the country's problems by attacking root causes.
So, we were pretty new at this, so we figured the best place to start
was to try to get our head around what we did and didn't know:
as a community, collectively, as a country.
What do we know about what does and doesn't work?
After all, philanthropists have been trying to save the country
and make the country better for many, many years
and have spent billions of dollars doing it.
Governments - state government, local governments -
have spent exponentially more on health care programs,
social programs, anti-poverty programs, job training, you name it.
So, we assumed that there had to be a massive body of evidence
that could help guide our investment decisions.
What we found was really alarming.
We saw bad research everywhere; it didn't matter where we looked.
For example, we turned to nutrition
as potentially an avenue to address our health epidemics.
We wanted to get smart about what factors and foods
might be conducive to chronic diseases: obesity, heart disease, diabetes.
And we found some very good research,
but we also found an abundance of studies like these:
"Chocolate makes you skinny."
Oh, "beer helps you work out," in case you didn't know.
"Coffee prevents Alzheimer's."
Super catchy conclusions.
A researcher notices that people in her study who are eating chocolate
are also skinny and summarily comes to the conclusion, or at least tells us,
that they're skinny because they eat chocolate.
Same thing with the Alzheimer's study.
Shoddy research, bad methodologies, small sample sizes,
correlations touted as causations,
selectively reported findings - just like the Amy Cuddy study.
By the way, there's a lot of these:
one might say a lot of "alternative facts."
(Laughter)
In 2012, a group of researchers randomly chose 50 ingredients
from a cookbook:
normal stuff like milk, eggs.
And they took a look at the body of research
relating to these ingredients to see what they could learn
about whether these ingredients did or did not cause cancer.
Seems like a worthwhile exercise; here's what they found.
For every ingredient you see listed in this slide -
wine, tomatoes, milk, eggs, corn, coffee, butter -
there was at least one research study
that argued that the ingredient caused cancer,
and at least one research study
that argued that the ingredient prevented cancer.
What are we supposed to do with this information?
We turned to health care: same thing.
We saw that the authors
of the vast majority of clinical trials reported in top medical journals
silently changed the outcomes that they were reporting.
So, they said they were going to study one thing, but they reported on another.
Now, why would they do that?
Look, I don't claim to be an expert.
Scientific research is enormously complicated;
certainly clinical research is, as well.
But one theory might be
that the original studies didn't pan out the way they wanted.
So, they cherrypicked positive findings from those same studies
on secondary outcomes and reported on those instead,
so they could get published.
Now, you don't have to have a Ph.D. to start wondering
whether maybe there could be something fishy going on here.
And these shenanigans happen everywhere.
Take a recent project that we did: The Reproducibility Project.
We asked researchers to reproduce 100 psychology experiments
that had been published in top psychology journals in 2008.
So, go do them again.
If you do them again, will you find the same results?
That's what we wanted to know.
You know how often they could find the same results?
One third to one half of the time.
Now, I'm not claiming that scientists and researchers
are actively and intentionally committing fraud.
I'm saying there's something broken
in our system
where scientists are feeling the need to report only positive findings
at the expense of the whole story.
How would they do that?
Well, lots of ways.
Let's say that a researcher wants to prove
that there's a relationship between eating blue M&Ms and prostate cancer,
because that would be newsworthy.
So, he designs and conducts an experiment,
but he doesn't find a relationship.
Okay, so he does it again: same experiment.
He does it again: no relationship.
He does it 17 more times:
no relationship.
But on the 20th time, he does find a relationship.
Boom - he can publish:
"Blue M&Ms cause prostate cancer a new study shows."
He doesn't tell us about the 19 other times
he conducted that same study and failed.
He just tells us about the one time that he succeeded.
That's called a "file drawer effect."
The researcher could also tweak the statistical anaylsis
until he gets the results that he wants.
That's called "P hacking."
Or he could report as to a very narrowly defined group
when in fact the original research study
was meant to address a much, much larger group.
Why does this happen?
Well, there could be lots of reasons,
but I would argue that a reason
is that the incentive system in science and in research is broken.
In an ideal world,
scientists and researchers would be motivated by one thing:
the pursuit of truth.
But in the real world,
scientists and researchers
are equally motivated by the desire to publish
because that's the vehicle for achieving tenure,
that's the vehicle for getting funding and for achieving notoriety.
Scientific journals are clamoring for articles that report flashy results,
so that those articles can get cited.
They even have a term for this: it's called the "impact factor."
Then of course, there's the media
and we, as consumers,
we all want to hear those flashy results,
and so researchers deliver,
even if it's at the expense of full transparency and rigor.
And we're all the worse off for it.
At our foundation, we try to break this cycle and reform the system
by funding organizations that are promoting transparency
and good practices and collaboration and data-sharing.
The Center for Open Science,
the Center for Evidence-Based Medicine at Oxford,
the Metrics Center at Stanford,
health news organizations that are holding the media's feet to the fire
as to what they report on research studies.
And they're all doing terrific work, but we need to do so much more.
We didn't become philanthropists
to hang out with academics and roam the hallowed halls of universities.
We became philanthropists to change the world.
But how can we even think about what to change
if we don't know what works?
And how are we supposed to know what works if we can't trust research?
So, this is our problem; this is our issue.
This goes to the core of who we are and what we do.
We can't function if we don't have a healthy research ecosystem.
This problem of bad research and bad science isn't limited to academia.
This is just as bad in public policy.
Anyone remember the "Scared Straight" program?
You know, at-risk youth would go to maximum security prisons
or meet prisoners,
and the prisoners would yell at them and tell them about their bad choices
and how they had to live their life on the straight and narrow?
We spent millions of dollars on these programs,
and they made intuitive sense except that they didn't work.
Research showed that these programs actually increased the likelihood
that these kids would commit criminal acts.
We should have known this much earlier.
We would have saved millions of dollars
and, more importantly,
maybe we would have saved some of these kids.
Three federal departments -
the Department of Labor, HHS, and the Department of Education -
have funded randomized, controlled trials
on social programs that have either been administered by the government
or by the private sector.
Now randomized, controlled trials are the gold standard in research.
Here's what they found:
70% of the employment and job training programs
that the Department of Labor looked at had weak or no positive effects.
Of the 28 teen pregnancy prevention programs that HHS looked at,
only three were worthwhile.
And 80-90% of the education programs that the Department of Education looked at
had weak or no positive effects relative to what schools were doing already.
So we know so little about what works in education, and in job training,
and in policy in general, even when we think we do.
Take the federal government's "What Works" clearinghouse,
which was established as a resource for practitioners
to determine what works in education.
It purports to rely on rigorous research.
Let's look at this research:
There is a study that had only a few dozen participants.
There's another study that was conducted over the course of around 12 weeks.
Then there's a bunch of positive labels
on secondary outcomes that ultimately don't matter.
For example, the website labels as positive a reading preparation program
because research showed that after going through the program,
kids were able to recognize letters of the alphabet.
Well, you might say
that's a stepping stone to reading, so that sounds pretty reasonable to me.
And it does, except that the website doesn't tell us
that those same researchers studied that same program
and found that the program had no effects on kids' ultimate ability to read.
Well, isn't that what we care about?
So we're spending millions and millions of dollars on programs
that at worst don't work and at best, we don't know,
or we don't have sufficient data or we don't have reliable data.
So this needs to change; we need to stop.
As a philanthropic community, as a policymaking community
we need to stop funding these programs
and relying on this ecosystem that isn't giving us the results
that we need.
And I've got some ideas on how to do that, how to reform the system.
First, more randomized, controlled trials.
Those of us who work with governments,
for governments, collaborate with governments,
need to demand more randomized, controlled trials
so that we can get evidence and understand what does and doesn't work.
Second, we all need to follow the evidence;
this is on all of us.
Governments and philanthropists
need to stop funding what doesn't work and start funding what does work.
We need to hold our own selves accountable for these results and for this problem.
This isn't something that was imposed upon us;
this is something that we've created and that's on us to fix.
And third, we need better data.
We can't do research if we don't have healthy data systems
that are speaking to each other, that are harmonized,
that researchers can access to give us the answers that we need.
Now, here, too,
there are excellent organizations both within and outside the government
that are working on these issues.
Within government. The Commission on Evidence-Based Policymaking,
the Social and Behavioral Sciences Team,
are conducting randomized, controlled trials,
pushing an evidence-based policymaking agenda within government,
and they're doing terrific work.
In collaboration with state and local governments
and the nonprofit sector,
the Rhode Island Policy Innovation Lab, J-Pal, Results for America,
are all contributing to reforming this ecosystem
by either conducting randomized, controlled trials
or working at promoting evidence.
And thanks to the work of these organizations,
and many others,
we now have some answers.
We now know which programs
do move the needle on things that we care about
like child welfare, and education, and job training, and recidivism.
Armed with that information, aren't we in a better spot -
as people who care, as concerned citizens,
as philanthropists, as policymakers - to make good decisions?
We need to follow data.
We need to focus more on research,
and we need to demand that of governments and of ourselves.
Because policy ideas that truly are worth spreading
are the ones that work.
Thank you.
(Applause)