(slow upbeat music)
- Over the past years, large deep learning language models
with billions of parameters
have improved the state-of-the-art
in nearly every natural language processing task,
from machine translation to document summarization.
Self-supervised learning has expanded beyond language,
to include images and videos.
Joint semantic representations
now operate across modalities,
as multimodal pre-trained models
for generation and understanding,
which enable scenarios like image captioning.
Overall, connecting language with other domains
in these large pre-trained models
creates incredible opportunities for innovation.
My name is Nan Duan and I'm a Principal Researcher
and the Research Manager at Microsoft Research Asia.
Today, I'm going to share with you
how we are capturing the semantic connection
between natural language and programming language.
I personally believe that artificial intelligence
has the power to fundamentally transform
how we understand software development.
The coding intelligence field
seeks to help software developers
by leveraging AI to develop large scale pre-trained models
for code and natural language.
And this is exactly what we have been researching
at Microsoft Research over the last few years.
We recently created a system called CodeBERT
for multiple programming languages
to support scenarios like transformation between code-code,
code-to-text, text-code, and text-text.
Let me tell you what this can mean for developers.
When developers are confused about what to write next,
code completion systems can help
by automatically completing the following tokens,
given the context of the edits being made.
When developers want to implement Java code
with the same function of some existing part of Python code,
code-to-code translation systems
can help translate from one programming language,
like Python to another, like Java.
When developers are looking at an unfamiliar piece of code,
they can rely on code summarization
to provide them with a natural language description
of what the code actually does.
When developers want to find code
that was written by others with the same intent,
code search systems can help to automatically retrieve
all semantically relevant code,
given natural language queries.
And finally, they can automatically translate documents
into thousands of languages.
We trained CodeBERT on both bimodal and unimodal data.
Bimodal data means parallel data of natural language -
code pairs and unimodal means codes
without paired natural language text.
What we do in essence,
is to take a code-text pair as the input,
and then randomly mask tokens from both sides.
The pre-training task is to recover the masked tokens.
We then fine-tune CodeBERT parameters for natural language,
code search, and code to documentation generation,
achieving state-of-the-art on both tasks.
You can reproduce our experiments in our latest paper,
which is available through our GitHub repo.
Some of our research breakthroughs
are already being implemented
in the tools that you use every day, like Visual Studio.
But the field of code intelligence
is still in its early days
and there's still so much more that we can achieve.
My dream is to be able to empower any creator
to easily develop software in any language,
be it natural or programming,
with the help of an AI code programmer.
But together, there are many big challenges
we will have to overcome.
Among, was that the field of code intelligence
lacked a benchmark suite that covered a wide range of tasks.
We saw how having a diversified benchmark dataset
was significant for the growth of applied AI research,
like ImageNet for computer vision and GLUE for NLP.
But nothing existed for Code Intelligence.
So, researchers from my team in China,
in collaboration with the Developer Division
and Bing Engineers, created CodeXGLUE,
a benchmark dataset and open challenge
for Code Intelligence,
so that we can push the field's boundaries
in partnership with the community.
CodeXGLUE is a collection of code intelligence tasks
and the platform for model evaluation and comparison.
It includes 14 datasets
for 10 diversified code intelligence tasks,
covering the scenarios we discussed before.
To make it easy for participants,
we provide three baseline models to support these tasks,
including a BERT-style, pre-trained model (in this case, CodeBERT),
which is good at understanding problems.
We also include a GPT-style pre-trained model, which we call CodeGPT,
to support the completion and generation of problems.
Finally, we include an Encoder-Decoder framework
that supports sequence-to-sequence generation of problems.
We invite the community to participate
in these open challenges
to help us accelerate the progress being made
in the field of code intelligence.
Moving forward, we'll extend CodeXGLUE
to more programming languages and downstream tasks.
And we will do this all while continuing to push forward
pre-trained models by exploring new model structures,
introducing new pre-training tasks,
using different types of data, and much more.
Stay tuned for the incredible progress
in this exciting field -
Thank you for listening!
(slow upbeat music)