Transcriber: Nina Belikova Reviewer: Amanda Zhu
So, I’m Marzyeh Ghassemi, and I want to talk to you
about how open health data and AI can improve your life.
And many of you already benefit from AI.
Your photos get sorted, your emails get written.
It’s great.
But my question is, would you let an AI diagnose you?
It’s a real question.
How would you feel about that?
I’m seeing some uneasy looks here.
So, I have spent 15 years at Oxford, at MIT, at the Vector Institute
studying and learning about AI.
I now have a team at MIT, the best and brightest graduate students.
And this is what we focus on.
This is what we think about.
And I am giving this talk today for those who hold health data
or do health data policy.
I’m here to tell you
that we can’t move forward in this space without open data.
Okay?
And the reason why is it used to be that progress could be measured
with CPU speed or maybe now GPUs.
But this has changed.
To have AI improve health specifically, we need large, diverse datasets.
They give us some understanding of what it means,
like, for a global population to be healthy.
Because right now what happens
is we take small datasets of people who are sick,
and then we predict what it would look like
if they got even sicker.
And we can do this really well.
So right now, AI performs at or above humans
in many tasks across the range of the human lifespan.
These are real examples, and that’s really exciting.
But what if I took one of those examples
and told you that the data that was used to train the model
was made by medical residents, not area experts.
Would you still want to use it?
Or what if I took one of those performance measures
that’s a really high number in aggregate,
and I told you that actually it performed significantly worse for women than for men
when you look at the subgroup breakdown?
And that’s why it’s so important to look at data.
Because all of these models, what are they learning from?
They’re learning from us.
Because our data is human generated, doctors are practicing,
and then a neural network is learning how to practice from a doctor.
But doctors are humans, and humans make mistakes.
And if a human makes a mistake
and we train an AI from that, the AI will too.
Garbage in, garbage out.
But it’s not like the garbage is distributed equally.
We know that many people now are saying, “AI will kill you.”
When I go to dinner parties, people hear that I do AI,
and they say, “Is it coming for us?”
But the thing...
You think I’m joking, but they do.
My children’s parents ...
My children’s friends’ parents will say, “Is it coming to get us?”
And the thing is, AI might be coming to get us in 100 years,
but what if doctors are killing us right now?
And we already know, it’s well known
that many subgroups experience worse care.
They die disproportionately more from conditions that they shouldn’t.
And so here is a place
where I think AI can actually help.
This is something that we can fix.
My group is working on healthy ML, Machine Learning, AI.
We’re trying to make models that are robust, private, and fair.
There’s an opportunity now in AI and health
unlike what we’ve seen before.
And as we’re working
towards making these robust, private, and fair algorithms,
we’re finding that it’s not GPUs or algorithms that are holding us back.
It’s data.
That’s where we have a lack.
And so let me talk to you through an example
of a patient like Sumana.
Maybe I want to improve her treatment through AI, right?
I want to take this patient and look at her needs.
If Sumana is having trouble breathing, and she comes into the hospital,
what is the first thing you might do?
You want to plan care, right?
I want to know what interventions she needs.
Does she need a vasopressor, or does she need a ventilator?
And the way that this is often done for many conditions
is we use checklists.
And checklists are great.
They’re ubiquitous in medicine because they’re easy to use,
they’re easy to deploy, and it’s easy to verify.
I can look at that list and say,
“It makes sense that I’m checking off these three factors,
and that means Sumana needs a specific intervention.”
But the way the checklists are made is by panels of experts.
And that has two problems.
The first problem is you have to convene a big panel of experts.
And that can take a long time to come to consensus.
The second problem is that experts are humans,
and humans have biases.
And so it’s been shown that many of the risk scores
that are currently used across medicine
perform significantly worse for specific subgroups -
for women, for minorities.
And that’s not something that we want to replicate.
So we actually just published a paper
that learns to create optimal checklists from data.
We used a mixed-integer programming approach.
And this is a great format
because it allows you to specify the model form,
model performance constraints,
and also things like fairness constraints.
And this is not something where a bunch of experts come in
and discuss what they think in the past they’ve seen contribute to a condition.
We actually use real ICU data to learn a real list,
and then we can add in the fairness constraints.
Let's walk through a real example of this.
Let’s say I want to predict mortality
after a renal replacement therapy in the intensive care unit.
It’s a real task.
It’s actually something you would want to do.
But I also want to make sure that I’m doing this fairly, okay?
So, the first thing that I might do is just run the model
and create a checklist.
And when I create this checklist,
you can see none of those boxes are referencing gender or ethnicity.
If I was an expert looking at it, it checks out, it is fine.
I checked it with real doctors.
But, if you audit this model on real ICU data to see how it performs,
it does significantly worse.
The maximum gap between the best and worst performance in subgroups,
between white men and black women is 53 percent false positive rate.
That’s crazy.
That’s really high.
But then if we add fairness constraints to the model,
now we’re not seeing the same large gaps,
because I’ve constrained the model to perform within that space.
There is one problem with adding in constraints.
Any time you constrain a model
- so looking here between what you do with and without constraints -
you’re going to lose performance somewhere.
That’s just what happens when you add constraints.
You can’t have everything you want.
This is no accident.
Fairness and utility are actually often at odds in AI.
You can't max them both out.
And what’s even more concerning is the same thing is true about privacy,
which is something we all care about.
And so when we’re making decisions about AI in these settings,
you can’t just max out:
“I want all the privacy, all the fairness and all the utility.”
You have to make real choices,
even if you take the state of the art differential privacy.
Differential privacy is added to AI models
to ensure that no adversary can identify some data point that’s too unique.
and that’s intended to protect the data from potential misuse.
But what we find is when you add differential privacy
to models that predict mortality,
you degrade model performance, the utility, so much
that it is no longer a model you would feel comfortable deploying
in the hospital.
And perhaps even worse, it’s not felt equally again.
So, we also find that adding differential privacy
removes influence from black patients.
And this is not something that we can fix algorithmically.
You have to collect a large amount of diverse data to fill a gap like this.
Otherwise, it’s a trade-off.
And until now, I’ve been talking about how we need more data from patients
if we want to improve fairness, utility, and privacy.
But there’s one other area, huge area, where we have very little data,
and that’s on caregivers.
Because in the end, it’s going to be a caregiver
who’s taking this AI advice and deciding what to do.
But we actually have very little evidence
of how best to interact with caregivers in hospital settings using AI,
which will make mistakes sometimes,
so that we don't make care worse if we make a mistake.
So, we decided to run an experiment to investigate how humans interact with AI
in a mental health emergency setting.
So, we showed participants call summaries to a mental health crisis line.
And then we asked them to choose between calling for medical help
and calling the police.
Some people were also provided with advice
from a large language model, an AI trained on human language.
And that was either biased or unbiased.
Here, a biased model was significantly more likely to suggest police
if the patient was Black or Muslim.
And I should say that we ourselves biased this model.
It’s very easy to do.
All you have to do is feed it biased input examples.
And so we fed more examples
of calling police when you see Black or Muslim
and fine-tuned this GPT-2 model
on just 2.000 extra examples of biased sentences.
And then we’re going to allow it to give recommendations in our experiment.
So, it’s this evil AI that we’re unleashing upon participants.
And we tested this in two ways.
This is the important part.
Respondents either received a prescriptive AI recommendation,
where we told them,
“We think the AI thinks that you should call the police,”
or “The AI thinks you should call the doctor,”
or we gave them a descriptive flag,
where we said that this has been flagged for violence or risk of violence.
Do you think that the framing of this biased AI
- it’s just as biased, just as likely to give you this violent flag
or say, Call the police -
do you think the framing will affect how clinicians and non-experts
adhere to the biased model advice?
Yeah, it does.
This is great news.
This is one of my favorite findings.
Because it means if you describe your biased model’s recommendations
rather than giving it prescriptively,
our subjects are able to retain their original fair decision-making.
This is really important.
It’s an exciting finding.
Because in this space, there are no easy answers.
The next wave of innovation, an ethical AI in health,
is really going to require robust thinking.
We need to collect data diversely,
we need to develop fair models,
and we need to deploy responsible recommendations.
And if we can do that across all of the settings I’ve spoken about,
I think we can actually move forward with creating actionable insights
in human health.
Thank you.
(Applause)