(mellow music)
- Hi, I'm Siddhartha Chaturvedi,
I work in health innovation at Microsoft,
and through my journey over the last 10 years
across the United States, the United Kingdom, and India,
I have realized how important NLP has become
as a major frontier of artificial intelligence.
But majority of the investments in NLP
have predominantly been focused
on Newswire PR or the general domain,
whereas high value, high volume domains have been left out.
To combat that deluge of information
that is coming in these high volume domains,
we think that there is a new approach that's needed.
And to that end, we've created
what is called Biomedical NLP.
To talk more about this,
I would love to introduce you to Dr. Hoifung Poon,
a Senior Director with Health Futures,
who's dedicated his last decade of work
towards Biomedical NLP and the applications of it.
- Hey everyone, I'm Hoifung Poon.
As Sid mentioned, I've been essentially
doing Biomed NLP research since my PhD.
Biomed NLP can already be applied
to PubMed scale knowledge extraction,
and support various applications such as question answering.
Biomedicine is at the cusp of a major transformation,
and Biomed NLP can play a key role in it.
The disruption from new technologies has made it possible
to attain high definition patient data at scale.
For example, the cost to sequence a human genome
has just dropped below 1000 bucks,
making it broadly applicable to cancer and other disease.
Meanwhile, regulatory changes have drastically
accelerated the digitization of patient data,
leading to the universal adoption
of electronic medical records or EMRs.
This disruption herald the new era of precision medicine
where treatment is tailored to individual patients
based on their fine-grained characteristics.
However, to fully materialize this potential,
there are major hurdles to overcome.
Every minute I'm speaking here,
two new papers are published in biomedicine.
Every day there are 4,000 new papers,
and every year over a million.
Biomed NLP can be applied to literature machine reading
and help unlock this ocean of fast-growing knowledge.
For example, we can quickly scan PubMed
and extract COVID knowledge graph
that uncover the relation between symptoms, the virus protein,
the immune pathway, or potential treatments.
Similarly, Biomed NLP can also be applied
to clinical machine reading, and help unlock
longitudinal patient information from the EMRs.
This is actually a key underpinning of this emerging field
of real world evidence that can help advance clinical care
and substantially accelerate clinical research.
In the mainstream NLP, the focus is on Newswire, the web,
which are very different from biomedical texts.
In fact, if you take a look
and compare the vocabulary of various BERT models,
you'll find that biomed terms
often are relegated to second-class citizens
in the general domain models.
A word like lymphoma may be shattered
into arbitrary software,
which make it very difficult to acquire the meaning.
In biomedicine, we also often need to combat
much more complex linguistic phenomenon.
For example, the majority of information extraction work
actually focus on binary relation within single sentences.
In biomedicine, however, we often need
to extract much more complex relations
that may span across multiple paragraphs.
For example, whether a mutation in a gene
may confer resistance to a drug.
More recently, we have developed
multi-scale representation learning,
which derived mentioned-level representation
for a partial relation from the local text spans,
and then combine them into a global representation
for extracting complex relation across paragraphs.
Finally, Biomed NLP also face
additional machine learning challenges.
Standard supervised learning require label examples
that in biomedicine, labeling such examples
require deep domain expertise,
which make it difficult to apply crowdsourcing,
which is often done in the general domain.
To overcome the annotation bottleneck, we have been focusing
on self-supervised learning for Biomed NLP.
Essentially, we are trying to hallucinate
a lot of noisy training example
using the domain knowledge applying to unlabeled texts.
We have developed a unifying framework
called "Deep Probabilistic Logic"
that can combine all of this noisy
and potentially conflicting sound supervision.
In principle, we're using probabilistic graphical models.
Most recently, we extended Deep Probabilistic Logic
and proposed self-supervision,
which can essentially learn new self-supervision rules.
So think about it as like hallucinating
a new hallucination scheme.
We also start exploring domain-specific pretraining
for Biomed NLP.
Here, we make two observations
that make us question the standard practice
to deal on top of general domain models.
First, as Sid mentioned, biomedical text
is very different from general domain ones.
Also in biomedicine, we actually have ample amount
of unlabeled texts that are available for pretraining.
Consequently, we have been pursuing an alternative paradigm
to pretraining from scratch entirely on biomedical texts,
for example, PubMed papers.
The resulting PubMedBERT model
actually outperforms all prior language models
in a wide range of Biomedical NLP tasks, which is promising.
For end application, we envision
an approach of human-computer symbiosis
in the form of assisted curation.
Biomedical machine reading can reduce the giant haystacks
such as the 4,000 paper for today
into a small set of candidate needles,
which the human curator can quickly verify
by essentially clicking a button.
In the literature machine reading,
we have successfully trained
accurate machine reader for precision oncology
without using any label examples.
We conducted in-depth user study
with an NCI designated cancer center.
For an example drop, their standard process identified
over 800 papers that are potentially relevant.
And normally, that will require them
over 1000 hours to actually curate them all.
Our Hanover system instead summarized it,
all this paper instantly,
and produced about 40 candidate facts,
which the expert curator can actually verify within an hour,
and already find over and automatically curate more facts
than they have ever curated in the past.
In clinical machine reading,
we have also made significant progress,
and can now extract key attributes for cancer patients
from the EMRs with very high accuracy.
And now we are starting exploring more general scenarios
in real world evidence and precision health.
We have released our PubMedBERT models,
and they have been downloaded
for over 25,000 times in the past month.
To accellerate progress in Biomedical NLP,
we also have created the resource of the BLURB,
which houses the first Biomed NLP leaderboard
based on the comprehensive benchmark
that span six tasks and 14 data sets.
In the BLURB website, you can find links
to download the PubMedBERT model from Hugging Face.
To load our model, essentially it just takes
one line of Python code.
You can also find instruction to download
and pre-process various data sets
and also submit to our leaderboard.
For more information, you can check out our BLURB website.
You can also find additional information
about our research agendas
on our Biomed NLP and Project Hanover websites.
You can also check out our MSR AI blog on our JAX partnership.
We are looking forward to see what you will find
with these models and datasets.
Thank you.
(mellow music)