(electronic upbeat music)
As humans, we can accomplish a variety
of intelligent capabilities,
like vision, language, science, and engineering,
using just one neural organ called the Neocortex,
and a single learning paradigm,
known as Prediction learning.
For a long time, different artificial capabilities
employed different model architectures
and learning approaches.
Looking forward, we seek to create unified architectures
and learning approaches across different AI capabilities.
My name is Han Hu, and I am a Principal Researcher
and a Research Manager in Visual Computing Group
at Microsoft Research Asia.
My research focuses on cutting-edge technologies
for computer vision.
With my team, we aim to build vision components
for the generic AI system.
In previous years, the team has developed several well-known
and widely used innovations,
including ResNet, Faster R-CNN, deformable convolution,
and High-Resolution Networks.
In this video, I will introduce two new series of works,
which have largely promoted the modeling and learning
unification between two most flourishing AI fields,
computer vision, and natural language processing.
Specifically, one series is about a new model architecture,
named Swin Transformer, which for the first time
demonstrated that transformer architecture
can significantly outperform previous
dominant convolutional neural networks
on broad vision problems.
The other work is a New Learning approach, named SimMIM,
short for "A Simple Framework for Masked Image Modeling."
Let's look into the first work, Swin Transformer,
which is a key work towards modeling unification
between Computer Vision, or CV,
and Nature Language Processing, or NLP.
It won the best paper award
of the 2021 International Conference on Computer Vision.
For context, the fields of CV and NLP
had evolved historically in a siloed way.
In computer vision, Convolutional Neural Networks
have dominated the field for more than 30 years.
But its template matching nature
makes it exponentially inefficient.
For example, we need three channels
to encode three slightly different head poses.
While developing a theoretically
compositional efficient backbone architecture
is needed and important, it's also so difficult
that all previous attempts
failed to be practical or mainstream.
We have long bet on Transformer
and its building block of attention
for basic visual modeling.
We hope its compositional modeling ability
to help encode these slightly different bird head poses
by only one channel.
More importantly, we believe its potential
to unify different fields.
We hope its general modeling capability
can make it a general architecture
for almost all kinds of AI capabilities,
including vision, language, social, and etc.
Over the last few years,
NLP methods evolved greatly, thanks to Transformer,
which has become a mainstream method of NLP modeling.
We've been working on how this approach
fits into vision problems.
Our team developed Transformer-like architecture
for computer vision problems three years ago,
by replacing all convolution layers in a ResNet
with Transformer layers, called Local Relation Network.
However, it is impractical at the time
due to slow inference speed.
Another well-known work is Vanilla Vision Transformer,
which directly applies the standard Transformer architecture
used in natural language processing
to the problem of computer vision.
We argue that this Vanilla Vision Transformer
doesn't consider the key difference
between visual and text signals.
First, a visual object can be in different scales,
which advocates a hierarchical structure
in architecture design.
A fluffy cat is a fluffy cat no matter how large it is.
For a sentence, the words or tokens have no scale variation.
We cannot say I am a fluffy fluffy cat cat.
Second, the visual signals are usually smooth in space.
For example, these two grass patches have similar texture,
and they are both colored greens.
This property encourages a locality prior.
In contrast, there is no spatial smoothness in a sentence.
Thirdly, a visual object can be in different locations,
which advocates translation invariance
in architecture design.
A fluffy cat is a fluffy cat
no matter where it is in an image.
In contrast, a word is sensitive to its location.
It has a different part of speech
when appearing at different locations.
By using these computer vision priors, we can build a bridge
from the Standard Transformer Architecture in NLP
to that in Computer Vision.
By introducing priors of hierarchy,
locality, and translation invariance
into the Transformer architecture,
we came up with Swin Transformer,
which performs strong on various vision problems.
An important technical innovation of Swin Transformer
is a Shifted Window approach.
This design ensures our fast speed in real computation,
particularly, when compared to the traditional
Sliding Window approach used in our previous work
of local relation network.
With the Sliding Window method,
information was collected of neighborhood pixels,
that is, each pixel is centered on itself
and takes a neighborhood window for calculation.
To improve speed, our team thus proposed a design
that doesn't overlap the windows
and exchanges information across different windows
by shifted windows; we proposed the next layer
to shift the window calculation by half a window
so that information can be exchanged
between different windows in the previous layer.
We hope the design paradigm provided by Swin Transformer,
combining Generic Transformer Unit
with the Domain-Specific Priors,
may inspire a wider range of fields
such as speech, science, engineering, and more.
Now, let me introduce a new visual learning approach,
SimMIM, which stands for
A Simple Framework for Masked Image Modeling.
This is one of our efforts towards unified learning
between computer vision and nature language processing.
In computer vision,
previously, we heavily relied on supervised learning.
For example, we'd learn visual features by a 1000-way
image classification problem using millions of images.
Thus, we proposed SimMIM.
It conducts visual learning in a self-supervised way,
based on masked signal modeling.
For each image, we mask a large portion of image patches,
and train the vision network
to predict these missing values, and so a model is learnt.
Please look at our side panel demo
to learn more about the concepts
regarding masked image modeling.
While prediction learning in NLP such as BERT and GPT
has been very popular,
it is important to carefully examine the difference
between vision and language signals
to make it work well for Computer Vision also.
Let's first look at the NLP and training task on the left,
where we will cover up the word "fluffy."
The goal of learning is to restore this word.
We notice that text has no spatial smoothness,
as the text tokens are semantically high-level and discrete.
In contrast, as you see on the right,
an image is highly correlated in space,
and the signal is raw, continuous, and low-level.
Regarding these differences,
we chose to use a large portion of masking area,
for example 60%, instead of a small ratio of 15%.
This adjustment to use a large portion
and a large ratio mask was a key to the success
of this SimMIM approach.
By this new approach, together with several works
of the same line from Microsoft Research Asia,
named BEiT and PeCo,
the learning approach between NLP and CV is also converging.
Empowered by Swin Transformer and SimMIM,
we trained a three billion dense vision model,
which is the world's largest visual model.
It has set new records on 4 indicative vision benchmarks,
including ImageNet image classification v2,
COCO object detection, ADE20K semantic segmentation,
and Kinetics-400 action recognition.
I hope you enjoyed learning about Swin Transformers
and SimMIM today.
We are excited about the future
of modeling and learning unification between CV and NLP.
To dive deeper into this work,
please visit the following links.
The GitHub for Swin Transformer and also for SimMIM.
And our recent paper titled Swin Transformer V2,
scaling up capacity and the resolution.
Thank you, and goodbye.
We have to actually rethink the entire AI stack.
What is Reinforcement Learning?
Let's look at an interactive demo.
All our code is now open source.
(upbeat music)