[upbeat electronic music]
Alan: Hi, I'm Alan, and I lead a research team
exploring AI's potential for improving healthcare.
Over the past five years at Google Health, our research has shown that
AI can augment a clinician's ability to detect breast cancer.
It can help people better understand their skin conditions,
and it can help researchers sequence genomes more accurately than ever.
Today, we are publishing research in Jama Network Open
on how AI can uncover brand new medical knowledge.
Here, our AI research revealed a tissue morphology feature
that predicts the survival of patients with colorectal cancer.
And clinicians in this research used this feature
to derive new insights for their patients.
Our work in this field has taught us that AI on its own
cannot solve all of healthcare's problems.
Medicine, after all, is about caring for people.
Data and algorithms must be combined with language and interaction,
empathy and compassion.
What makes us healthy is complicated.
It's specific to geography and it's influenced by social drivers.
We believe it is imperative to actively work
to include diverse experiences, perspectives, and expertise
when you're building AI systems.
To bring this vision forward,
we're exploring how AI models in medicine
can use language and interactivity
to be more effective, more helpful, and safer.
Late last year, we took our first step towards
rethinking conversational AI systems in medicine with Med-PaLM,
a large language model designed to provide
high quality and authoritative answers to medical questions.
We built Med-PaLM by instruction prompt tuning PaLM.
PaLM is a 540 billion parameter large language model from Google Research.
We did this work with a small set of carefully curated
medical expert demonstrations.
In multiple choice questions used for US medical licensing exams,
the pass mark for new doctors is often around 60%.
These questions have long been considered
a grand challenge for AI systems.
They require a clinician to recall medical knowledge
and apply logic to identify the correct answer.
Despite years of effort from leading AI labs around the world,
performance on challenging tasks like this has plateaued at around 50%.
Last December, our model, Med-PaLM,
was the first AI system to exceed the pass mark.
We reached a performance of over 67% on these licensing exam style questions.
We also carefully examined how Med-PaLM performed in many other kinds
of medical question answering tasks.
These ranged from commonly asked internet search questions
to complicated questions about medical research.
In doing this work, we compared Med-PaLM's answers
with answers from real clinicians,
and we looked at several aspects like factual accuracy, bias,
and the potential for harm.
You can see one common question here,
and how Med-PaLM answers this question, which is about incontinence.
In this case, Med-PaLM's answer is generally sound,
but it's not as comprehensive as the answer given by the clinician.
The clinician here names multiple specific causes of incontinence,
as you can see,
where Med-PaLM is less comprehensive.
And doctors rating this answer from Med-PaLM agreed.
They found it generally accurate and safe,
but you can see here that they highlighted that
there's room for improvement in the level of detail our system provided.
You can see from this sort of work that we're still learning.
One interesting aspect of this kind of work
is that the evaluating physicians' rating of this answer
may also change depending on their own clinical expertise
and their experience in this subject area.
In this next example, Med-PaLM's answer is complementary to the clinician's.
Med-PaLM mentioned similar information, but in different ways.
And it makes this lovely point about how the severity of symptoms can vary
depending on the type of pneumonia and the overall health of the person.
Doctors rated this answer as very high quality across our rating framework.
We believe it's really important to innovate responsibly
by doing this kind of rigorous research in healthcare.
Today, we're announcing results from Med-PaLM 2, our new and improved model.
Med-PaLM 2 has reached 85% accuracy on the medical exam benchmark in research.
This performance is on par with expert test takers.
It far exceeds the passing score,
and it's an 18% leap over our own state of art results from Med-PaLM.
Med-PaLM 2 also performed impressively on Indian medical exams,
and it's the first AI system to exceed the passing score
on those challenging questions.
There are many ways that an AI system like Med-PaLM can be a building block
for advanced natural language processing in healthcare,
and we'd like to work with researchers and experts to advance this work.
The potential here is tremendous.
But it's crucial that real world applications are explored
in a responsible and ethical manner.
And as we do that, we're looking forward to sharing our progress
with you at a future checkup.
[upbeat electronic music]