--a lot of research showing that people learn best
when they, themselves, become teachers
and when they work interactively in a hands-on kind of way.
And also research showing that people learn well
with empirical feedback.
So we designed a project that combined all three
of those elements to help students become more
expert at debiasing their own, as well as others', judgments
and decisions.
We always began by having students complete a decision
exercise.
And in this exercise, there's usually
three people who are depicted.
And, for example, they might be three surgeons.
And each surgeon is described just the way
they would be on a regular website in real life.
So you see a little picture about this surgeon
and a description of the surgeon.
And then the participant or student, in this case,
has to decide which surgeon they would
like to see, imagining that they needed a particular kind
of surgery, like foot surgery.
Now, we, ahead of time, have inserted
key diagnostic information into the description of each surgeon
so that a person reading it can tell
which surgeon is likely to have the best success rate.
And that's what we call in decision theory
a diagnostic or normative cue.
It's the cue we want them to use,
it's the one that helps them maximize their expected value.
Then we also embedded a bunch of non-diagnostic distractor cues
that often lead people to make biased choices.
And these are things like what the person looks
like, their race, their class, their ethnicity,
their religion and, really, every judgment and decision
making bias we could possibly load into it.
Things like order effects, the order
in which different information appears.
And so, what we found is that most respondents ended up
choosing a surgeon who looked like the American stereotype
of what a surgeon looks like, a Caucasian male.
But, in fact, that surgeon did not
have the best expected value or success rate.
And so then we revealed the data to students
and they saw for themselves that they had fallen into this trap.
And what's nice about this exercise is that, regardless
of what your views are politically
about affirmative action or no affirmative action,
everyone agrees that there is a right way,
there is a right choice.
And that you shouldn't be using something like race
to make the choice, you should be using
who has the best success rate.
So that's just the first part of the task.
Once people see their own responses
and they see that the class as a whole
has made a suboptimal choice, they
feel highly motivated to want to correct it
and to want to teach others how to avoid it.
So that's when the real innovation comes in.
And, essentially, what we do is we give them the opportunity
to first, identify all of the biases
that are embedded within the task.
They have to name each one and define each one.
And then they have to design what
we call choice architecture.
It's a term borrowed by Cass Sunstein and Richard Thaler's
book, where it's just a small piece of information that
helps other new respondents to make
a rational choice or a choice that maximizes expected value.
Amazon Mechanical Turk is this online marketplace
where you have somewhere between probably 10 and 50,000
individuals a day who are, in this marketplace, who
are taking surveys of all different types.
Some of them are serving as validation for machine learning
algorithms, others are psychology experiments.
So ours is just one of hundreds, if not
thousands, that are on there that individuals
are paid to take the survey.
And that's broadly nationally representative
with a few differences.
Students are split into teams.
Each team, after listing all of the decision biases that are
embedded within the scenario, work together to create a kind
of choice architecture that will be used in a real-world
scenario-- actually run online on MTurk--
to try to debias participants in that activity online.
So they come up with a lot of different ways of doing this.
Sometimes they'll include some sort of table
to try to teach people how to calculate an expected value
through numbers.
Sometimes they'll tell people, don't consider
the race of a doctor, that shouldn't matter,
or the gender of a doctor, that shouldn't matter.
And they'll decide as a group which method they think
will be the best way to debias these people online.
And we'll try to coach them through that.
What the HILT money has enabled us to do
is systematize a whole set of aha moments.
Because there are a number of these moments as you
unfold the exercise with students,
where there's the first aha, where it's not such a good aha.
People are like, ooh, ooh, I made a bad choice.
And then there's the, I get to try to fix this.
And then there's the good aha, where
they see how well they did and then say,
oh, so this worked better than that.
And then there's even another aha,
which is they see what other teams did.
And so that's really, I would say, the third aha.
And then the last aha, actually, it occurs to me,
they don't necessarily see for themselves
but we try to, at a meta level, encourage them to understand
that they can use this kind of process
for testing their assumptions empirically
in whatever setting they're in.
And so we use this with executive education students as
well as Harvard enrolled students
to help them think about, they don't
have to just hold on to their existing beliefs about what
creates change, but actually submit them to empirical tests.
So we began this project before the HILT funding
in executive education.
And we had one scenario, we were having
participants choose a surgeon.
But we had this idea that, this isn't just
about choosing a surgeon.
This can be about any choice where there is a correct answer
that people aren't getting to.
And what HILT allowed us to do and the reason we applied
was we wanted to move this beyond executive education
to our degree program classes, where we have undergrads,
MD's, people from the School of Public Health, the business
school, and then test this across a whole host
of different scenarios, so not just surgeons.
And what HILT funding allowed us to do was test this idea
and see how it could move beyond just executive education,
in this one scenario, to a whole host of different types
of students and a whole host of different scenarios,
and really expand the exercise and grow it.
Without the structure that HILT brings
and the funding that HILT brings,
we probably wouldn't press ourselves to really
think hard about what are the key learning mechanisms here.
And what is exportable, what is generalizable,
what isn't, where are the weak points?
And we also certainly would never have had a coach to help
us have an outside perspective on it.