Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to have a look at DeepMind’s Flamingo AI that has
looked at over 1 billion images, and see what it had learned from it. And, yes,
we will also try to break it. Successfully. Now, its goal was to fuse AI models that understand
language with other AI models that understand images. And what comes out? Absolute magic.
For instance, it can be an amazing assistant. We can give it a picture, ask it what it depicts,
a bowl of soup with a monster face. I particularly like this one because this image was also made by
a different AI. So this is an AI commenting on another AI’s work. How cool is that?
Now, we can also subject it to a test that is interesting because it is designed to break
the brain of some humans. Look. We need to read the text and then name the color that was used
to write it. If you try to do it quickly, it is easy to answer it incorrectly, and apparently,
this little AI is passing with flying colors. But it gets better! Look! It knows that this
is the Stroop test, and it also knows that humans read this kind of text slower. So,
let’s pop the question: is it challenging for you too, little AI? “I am not affected by this
difference.” Whoa. It is flexing. But, it backs it up. What a time to be alive!
Now, what it can also do is something called few-shot learning. What is that? Well,
this is learning from only a few examples, something humans are really good at, but machines,
not so much. For instance, we look at an image, and say that this is a chinchilla. We look at
another image, and say this is a shiba. Then, the third time, we look at an image, and we say “this
is”. So, does it know what it is supposed to do with this? Remember, neural network-based learning
methods typically require thousands and thousands of training samples to learn something new, so,
can this new one do it? Yes! Oh boy. That is amazing. So, this is few-shot learning.
And I have a hard time overstating how incredibly useful it is. For instance,
we can quickly teach it to read handwritten text from a piece of paper, or, here comes my favorite:
we can even ask it to reverse-engineer the prompts that the AI used to create these images. We teach
it on 2 examples where we tell it the piece of text that was used to generate it, and now,
for the third time, it knows the drill and does it itself. Or, we can even ask for the ingredients,
or even the nutrients that are present in a meal, or what songs a soundtrack of a movie contain.
And, here is another reassuring thing. It does not
fall for the good old apple + iPod trick that many previous techniques fell for.
So now, let’s try to answer three Scholarly questions. Question number one,
is it really better than previous techniques? If we compare it to previous zero-shot techniques,
this is a challenging case where the AI has to perform something new, there is
no contest. The new one is so much better. However, what I found even more interesting
is that it can even give fine-tuned AIs a run for their money too. This is insanity. So,
what is that? Fine-tuned means an AI that was specifically designed and trained for one task,
and one task only. This new method can do many things at the same time, this is a generalist,
so it is not expected to beat specialized techniques. How could it? And here comes
the the insanity part: it is not only competitive with these fine-tuned techniques on several
datasets, but, wow, it even outperforms them on some of these despite having access to ten,
and sometimes even a hundred times less training data. I am stunned. These improvements are
concentrated around video-based question answering datasets like NextQA and iVQA.
Question number two, how does it improve with the number of shots,
you remember the flamingo example, we had two shots here. The answer is that it gets it early,
it does not need tens of examples to understand what we are trying to ask.
Its zero-shot performance is formidable, so it can do things it has never been taught before,
that is incredible, but even in other cases, after just a few examples, it really gets it.
And, three, the usual suspect. Model size. Yes! As I hoped,
results still improve if we train a bigger neural network, which is great news,
two more papers down the line and it will be so much better, even without algorithmic
improvements. If we increase the model size, it gets smarter, and here is a beautiful chart
that showcases that it can learn from only a few shots significantly better. Dear Fellow Scholars,
this is perhaps artificial intelligence being born right before our eyes. What a time to be alive!
But wait, we said that it is excellent at video-based question answering. So,
let’s push it to its limits and see what it can do. Whoa, that is very nice. It understands that
this good boy is being weighed, understands what a video game avatar means, reads a sequence of text,
even if only a few letters are shown at a time as we pan the camera through it.
And, hold on to your papers, because it also understands humor. Yes, that’s right.
This needs careful prompting, but it can do it. Here, we are looking at an image where
people are getting amused by Obama’s little prank with the scale. With a little guidance,
it successfully identifies what is unusual and amusing about this image.
But, we haven’t broken it yet. And now, you will see that not even this technique
is perfect. For instance, let’s try to mess with this little AI and break its brain. How
do we do that? Well, of course, with silly, irrelevant questions. We can even ask it what
it can see outside the window? And it says, a parking lot. Well, not quite! Also, whom is the
person texting? It says, of course, the driver. Well, you cannot possibly know that, little AI.
And once again, this is an incredible paper, perhaps a step towards the highly coveted
general intelligence that so many people think we will never reach. I am stunned. So,
what do you think? Does this get your mind going? Let me know in the comments below!
Thanks for watching and for your generous support, and I'll see you next time!