(Upbeat music)
- [Weizhu Chen] Hi!
I'm Weizhu Chen from Microsoft Azure AI.
Today,
I'm excited to share with you
one the Turing family of models,
DeBERTa and its achievement of
SuperGLUE human parity.
My team in Microsoft has been
developing large scale models to solve
some of the key problems in natural language understanding
such as LEM NT recognition,
machine reading comprehension,
and question answering on structural data.
We also use this large scale model
to power Microsoft products like Cognitive Services
in Azure AI and Dynamics 365 Business Applications.
As you know,
natural language understanding (NLU) is one
of the longest running goals in AI
and SuperGLUE is currently among the most challenging
benchmarks for evaluating NLU models.
This benchmark consists of a wide range
of NLU tasks including
question answering,
machine reading comprehension,
reasoning, natural language inference and others.
Take the original task COPA as an example.
Given the premise, I tidied up my house
and the question is
what's the cause for this?
The model is asked to choose the answer
from two candidates.
The first one is
I was swamped with work.
And the second one is,
I was expecting a friend.
Where it's easy for a human to choose
the right answer, it is often challenging
for the AI model to get the right answer.
The model needs to understand the causal relation
between cleaning up the house
and expecting the friends.
This kind of common sense, may be new to the model.
But because the model has learned so much
about human language, it is able to detect this kind
of logical connection.
Another task that the model is pretty good
at is machine reading comprehension.
For example, given a piece of text
the model can quickly answer questions
about the content.
You can see a lot of examples
in the side panel.
So we can say SuperGLUE is a great benchmark
to measure the progress of NLU models
Since its release in 2019.
Recently, we developed the DeBERTa model
with 1.5 billion parameters
in this innovation. In January 2021,
our single DeBERTa model
surpassed the human performance on SuperGLUE
for the first time.
DeBERTa is a transformer based
neural language model.
Pre-trained on lots of raw text
using self-supervised learning.
Compared to previous pre-training language models,
DeBERTa introduced three innovations.
Disentangled attention mechanisms
Enhanced mask decoder
and a new adversarial training measure.
Disentangled Attention is a new way
to capture the relative position of each token.
Unlike BERT where each word in the input
layer is represented using one single vector.
to encode the content embedding
and the positional embedding.
DeBERTa used two vectors to encode its content
and relative position separately.
We also applied the disentangled metrics to
capture the dependency between the content
and the relative positions
in the self-attention calculations,
As well as
proposing an efficient implementation for this.
The motivation for this is clear.
The relation between any two words should depend
on both their content and relative positions.
For example, the dependency between the word deep
and learning is expected to be much stronger when
they occur next to each other,
other than when they appear
in two separate sentences. The Enhanced Mask
Decoder is to learn how to incorporate
more information to help the model predict
the mask tokens.
Previous work can incorporate more information
in the input layers.
Using the Enhanced Mask Decoder
this new mechanism can be used to incorporate
different kinds of information
about tokens in DeBERTa.
We chose to incorporate the absolute positions
of our tokens.
Since it can be a complimentary information
to the relative positions.
We can see this motivation from an example.
Consider the sentence,
a new store opened beside the new mall.
With the word store and mall masked for predictions
the local contexts of these two words are very similar.
But they play different syntactic roles
in the sentence.
For example, the store is the subject,
but mall is not.
This, syntactical nuance highly depends
on the words’ absolute position in the sentence.
So it's important to account for a word
absolute position in the language model pre-training.
The last innovation in DeBERTa
is a new regularization method
for improving model robustness and generalizations.
The Virtual Adversarial Training introduced a controlled
perturbation in the embedding layers
and asked the model to produce
the same outputs. Inspired by
the layered normalization technique,
we develop a SiFT (scale
invariant fine-tuning)
where the perturbations are applied
to the normalized embedding layers,
instead of the original embeddings.
We observe these can address
the instability issues
and bring significant gain in model performance.
We have released
the DeBERTa model and the code to the public
and there are already a lot of great results
built on top of these models.
For example,
researchers from UC Berkeley have applied
the DeBERTa 1.5 billion model
on their legal contract review dataset
and achieved significantly better results
than previous models.
You can also apply the Microsoft
Deep Speed optimizations on the DeBERTa models.
So you can use one single GPU, like V100
or RTX3090
to fine tune the 1.5 billion DeBERTa model
for your own task.
DeBERTa was integrated
into Microsoft Turing model family,
to power Microsoft products
like cognitive service in Azure AI,
Microsoft Office business applications,
and Microsoft Dynamics 365.
This can benefit hundreds of millions of users,
through the Microsoft AI @ Scale initiative.
This is a joint work
with my colleagues Pengcheng He
Xiaodong Liu and Jianfeng Gao.
Thank you.
(upbeat music)