colleagues.
This paper introduces Vision Transformer, and tries to replace CNNs with transformer
architecture for computer vision tasks.
Transformer architecture revolutionized Natural language processing.
It has been used in many state-of-the-art models and has replaced the RNNs that once
dominated the field.
The reason for this gradual shift from RNNs to Transformers was that Transformers were
significantly faster than RNNs because they could do parallel computation, something RNNs
are incapable of.
The transformer architecture has been employed in a variety of applications.
Other than NLP, transformer has been used in speech processing and some other tasks
as well Image processing, on the other hand, was a difficult domain for transformers to
operate in.
The reason is, transformers heavily use self-attention and self attention is really computation-hungry.
Consider this simple 3-word sentence.
Using each word as the query, we should calculate an attention weight between that word and
every word in the sentence.
Which means for a 3 word sentence like this example, we will have to calculate9 attention
weights.
And for a n-word sentence, we will have to find n-squared attention weights.
As you can see, the computational complexity of self-attention is quadratic.
Now, if we want to do the same thing in an image, we should calculate an attention weight
for each possible pair of pixels.
Which means, for an ordinary selfie like one you can see here, which has about 8 million
pixels, we should calculate more than sixty trillion attention weights!
Even for simpler images, like hand-written digits in MNIST data set, which are 28x28
pixel images, we’ll have to calculate more than six hundred thousand attention weights!
We need to find out a way to reduce the amount of times we have to do the self-attention
operation if we want to use transformer for images.
to overcome this problem, researchers tried several methods.
For example, instead of applying self-attention globally, they applied it exclusively in local
neighborhoods for each query pixel.
Convolutions could be totally replaced by such local multi-head dot-product self-attention
blocks.
The vision transformer model presented in this paper seeks to be as close to the original
transformer model as possible.
In this model, we first split each images into fixed-sized patches We then take each
patch, Flatten it into a vector, And then use a trainable matrix E to transform flattened
image linearly, Resulting in a new vector that we call “patch Embedding” Along with
patch embeddings, We also create a new, trainable, [CLASS] token.
This token works just like the [CLS] token in BERT: it can be used to classify the image.
But before we send these embeddings into the transformer encoder, we should consider the
fact that transformer doesn’t know anything about the order and position of its inputs.
So, these two images in this slide are identical to a transformer.
Therefore, we have to somehow inform our model about the position of each patch.
Just like the original transformer, here we can simply add a positional encoding to our
patch embedding to include spatial information.
For each position, there is a learnable positional encoding vector.
We add it to the patch embedding and send the result to the transformer encoder.
This encoder is exactly the one described in ‘attention is all you need’ paper.
If you want to know more about it’s inner working, I recommend you watch my comprehensive
video on the paper ‘attention Is all you need’ in which I try to explain the transformer
architecture in detail.
One of the transformer outputs corresponds to the [CLASS] token and will be sent to a
softmax to classify the image.
The rest of the outputs, on the other hand, are not important and we can get rid of them.
So this is how you train your vision transformer.
If you’d like to fine-tune it for a downstream task you simply remove the prediction head,
And attach a zero-initialized feedforward layer instead.
As you can see here, Vision Transformers perform better than a very big, CNN-based network.
While being faster and more efficient in the terms of computational costs.
Transformer models also outperform CNN-based models on visual task adaptation benchmarks.
So, does this mean we no longer need CNNs and can use transformers for our computer
vision tasks?
Well, let’s not rush into conclusions yet.
As you can see in this figure, CNN-based models actually outperform every transformer model
when we pre-train them with small number of data.
And in order to have transformer models that can beat CNN-based models, we need to train
big transformers on enormous datasets.
So, if you are google, you’d better put CNNs aside and use this new model.
But if you are not google, you still have to work with CNNs. alternatively, you could
use pre-trained vision transformer model and fine tune it so it can perform well on your
downstream task.
In this Video, I first described one of the biggest flaws in attention mechanism; which
is the fact that it is computation-hungry.
Then I showed you how the authors of this paper got around this difficulty.
We learned about Vision Transformer, a large model that can be fine-tuned for a variety
of computer vision tasks.
And we observed that, given enough data, vision transformer outperforms CNNs.