TAKUMI OHYAMA: Hi, everyone.
I am Takumi, Machine Learning Engineer at Google Advanced
Solutions Lab.
Currently, a lot of people are talking about generative AI
and its new advancement.
And as some of you may know, Google and Google Cloud
also released so many generative AI-related new products
and features.
But in this video series, our goal
is not to create a state of our generative AIs,
nor to introduce Google Cloud new products.
Instead, we will explain what kind of technology
is working behind them.
And especially in this video, I'm
going to talk about how to actually create
a very simple generative model, image captioning
model, by using technologies like encoder-decoder, attention
mechanism, and a bit transformer.
If you're not very familiar with these concepts,
I recommend checking other videos talking about them
before this.
OK.
So if you're ready, let's talk about image captioning tasks
and data set at first.
We're going to use this kind of data set.
As you can see, there are a lot of pairs of images and text
data, and our goal is to build and train
a model that can generate these kind of text captions
based on images.
And we'll make it happen by building this kind of model.
As you can see, it is a kind of encoder-decoder model.
But in this case, encoder and decoder
handle different modality of data, which is image and text.
So we pass images to encoder at first,
and it extracts information from the images
and creates some feature vectors.
And then the vectors are passed to the decoder,
which actually build captions by generating words, one by one.
So this encoder part is easy.
You can use any kinds of image backbone,
like ResNet, EfficientNet, or Vision Transformer.
What we want to do here is to extract features
by using that kind of backbones.
So code is very simple, too.
In terms of the code, we're going
to see the entire notebook example in the next video.
So here, let's just focus on some important lines.
As you can see, we are using classical InceptionResNetV2
here from Keras applications.
But again, this can be any image backbones.
So the next part, the decoder, is a bit complex.
So let's take a look very carefully.
So this is the entire architecture of the decoder.
It gets words one by one and makes the information
of words and images, which is coming from the encoder output,
and tries to predict the next words.
So this decoder itself is an iterative operation.
So by calling it again and again autoregressively,
we can eventually generate text captions.
There are so many variations for this decoder design.
But here, we build transformer-like architecture,
although we still use RNN or GRU.
So let's zoom into each component.
The first embedding layer creates word embeddings, which
was discussed in other videos.
And we are passing it to GRU layer.
If you forgot what GRU is, it's a variation of Recurrent Neural
Network, or you can call RNN.
RNN takes inputs and updates its internal states
and generates output.
So by passing sequential data, like text data,
it keeps the sequential dependencies
from previous inputs, like previous words.
The GRU output goes to attention layer,
which mixes the information of text and image.
In TenserFlow Keras, we can use predefined layers
in the same way as other layers.
There are multiple implementations,
but here we simply use tf.keras.layers.Attention.
But if you want to use more transformer-like architecture,
you can also [INAUDIBLE] tf.keras.layers.
multhead.Attention, which uses multiple attention heads.
You can simply switch and use it in almost the same way.
Inside attention layer, it looks like this,
as you may have already seen in another video about attention
mechanism.
But a unique thing here is it pays attention
to image feature from text data.
By doing so, it can calculate attention score
by mixing both information.
Going back to code, you can see this attention
layer takes two inputs--
gru_output and encoder_output.
Internally, gru_output is used as attention query and key
and encoder_output as value.
The last components are add layer and layer normalization
layer.
Add layer just adds two same-shaped vectors.
As you can see here, gru_output is passed to attention layer,
as we discussed, and to this add layer directly.
These two flows are eventually merged in this add layer.
This kind of architecture is called
skip connection, which has been a very popular deep neural
network design pattern since ResNet.
So it is also called residual connection.
This skip connection is very useful, especially
when you want to design a very deep neural network.
And it is also using the transformer.
With this, now we could build inside our decoder.
So we are ready to train the encoder-decoder image
captioning model using the captioning data set.
We will see how it works in the next video.
But before moving onto the next one,
I want to explain a bit more about inference phase,
where we can actually generate captions for our visual images
because this process may look a bit complex.
Here, you can see three steps, and we're
going to implement these steps in a custom inference function.
The number one, generate the GRU initial state
and create a start token.
In training phase, TensorFlow Keras
can automatically handle gru_state for each sequence.
But in this inference phase, since we
designed our own custom function,
we need to write a logic to deal with it explicitly.
So at the beginning of each captioning,
we explicitly initialize the gru_state with some value.
And at the same time, remember, our decoder
is an autoregressive function.
But since we haven't got any word prediction yet
at the beginning of the inference,
we pass start token, which is a special token that means
the beginning of a sentence.
Number two, pass an input image to the encoder
and extract a feature vector, as we discussed.
And number three, pass the vector to, this time, decoder,
and generate a caption word in the for loop
until it returns end token or it reach to MAX_CAPTION_LENGTH,
which is just a hyperparameter specifying some number,
like the 64.
And in this for loop, we define all the procedures
of caption generation by calling the decoder autoregressively.
End token is a special token, too,
which means the end of the sequence.
So when our decoder generates this token,
we can finish this for loop.
Or you can go out of the loop when the length of the caption
reaches some number, MAX_CAPTION_LENGTH.
Let's take a look at the code one by one.
In the first step, we initialize two things--
gru_state and start token.
In this case, gru_state is simply
initialized with zero vectors.
And we set start tokens as the first input word
for the decoder.
In terms of the word_to_index function used here,
I'm going to explain in the next video.
But basically, it's just tokenized words
to word token, which is the standard text preprocessing
technique.
In the next step, we preprocess the input image a bit
and pass it to the encoder we train.
In terms of the image preprocessing,
it reads in decode_jpeg in the first line.
And it resizes it from any arbitrary resolutions
to a specific resolution.
And it changes the scale from 0 to 255
to 0 to 1 in the third line.
In the last phase, decoder for loop, it is a bit complex.
So I will explain, in the next video,
more in detail with the actual code.
But the main thing here is to call the decoder
by passing three things.
dec_input means decoder input, which
should have a word token predicted
in the previous iteration.
And as we talked, if it is the first iteration,
this would be the start token.
gru_state is the current GRU state we discussed.
And please note that the decoder always
outputs the updated GRU state.
And last but not least, features.
This is the image feature we extracted with the encoder.
By passing them, we can get the actual next word of prediction.
This is a very simple text generation model from images.
But this kind of iteration is very similar,
even in very large, longer generation models,
like Googlebot.
They basically predict the next word autoregressively
in this way, one by one, based on some information
and learned knowledge, which is embedded
in a huge number of parameters.
In the next video, I will walk you
through the entire notebook, and then we
will check what kind of captions this model can generate.
Thank you so much for watching, and see you in the next video.