So while we are all stunned by OpenAI’s new GPT-4o, Google also
held their own keynote with lots of new goodies.
And one of the highlights of the show was Sir Demis Hassabis unveiling Project Astra,
a universal assistant that is with you all the time.
You can ask what that part of the speaker is called draw an arrow,
and it will tell you this. You can point it to a piece of code and ask it what it does
and it will tell you. You've seen this feature in OpenAI’s GPT-4o as well. Awesome. In fact, Gemini,
Google’s AI has also watched the OpenAI event and it can even answer your questions about it.
Now, back to Project Astra. This one has an amazing feature. What is it? Well,
it remembers. When you walk around in the office,
it remembers where you put your glasses before and can tell you.
Or, it can do this too.
Some say that this is the future of AI at Google. And if I had any doubts
that smart glasses are the future, I don't really anymore. And there is so much more.
Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Now, I mentioned that this is an AI that remembers. How does it remember exactly? Well,
their new Gemini 1.5 Flash AI has something that
we call a wide context window with 1 million tokens. What does that mean?
It means that you can not just prompt it with text,
but can upload your entire thesis in there, with videos of talks and everything and
ask it to role-play as your thesis committee and ask you tough questions to challenge it.
It can watch movies and videos too, for instance, I made it watch one of my favorites,
Jacob Collier. I gave it a question about an almost 10-minute video that is in an
unnecessarily high resolution, this came out to about a 160k tokens, and it takes about
thirty seconds to get an answer. It is not perfect, but it is very impressive that it
is close to it. Thank you Google DeepMind for the opportunity to play with this.
The previous 1.5 Pro version also had a similarly wide window, we talked about the fact that it
has a quadratic computational complexity and that means that it can take a minute,
or even longer to answer. And the new Gemini 1.5 Flash is promised to be able to do this much,
much faster. In fact, remember how fast GPT-4o was? Well, the first benchmarks show that it
might be close to twice as fast. Saying that it is blazing fast would be an understatement.
There will be an update to its open model version too that you can run locally for free,
it will be called Gemma2 and will come in a 27b parameter package,
that is a good size to run on a beefy desktop computer, and likely there
will be smaller versions too that you can even run on your phone, like Gemini Nano.
Now, they also showed a sneak peek of Imagen 3, their text to image AI. You know the drill,
you write a piece of text, and you get an image, but this time with more details,
and they also promise better text. That’s good because text is still
something that even the state of the art systems struggle with.
But, in the age of OpenAI’s Sora, text to video is also possible, and to compete with it,
they showcased Veo, which creates Full HD videos. Up to a minute long each. Super long. Now note
that Google DeepMind publishes a ton of papers, but this event was not showcasing peer-reviewed
papers, it is more like a sneak peek into the new things they have in the works. And here,
we are focusing on things that were or will likely end up as papers. For instance,
Veo is a culmination of a lot of earlier papers, Phenaki, VideoPoet, Lumiere, many of which we have
talked about here in the series. Yes, in this corner of the internet, you get to become a
Fellow Scholar and see the history of artificial intelligence unfold right before your eyes.
Now I also see long-term temporal coherence, which means that it remembers what it did a moment ago,
so the car will be the same, the environment will be the same if we look away and look
again. I think the visual quality is still a bit behind OpenAI’s Sora, and my impression is that
they will try to compete on creative control tools instead. That is a good direction too.
Note once again that this is a keynote, not a peer-reviewed research paper,
so it is harder for me to find potential faults, cherry picked results and so on.
This is a disclaimer that I would like to draw a little more attention to in cases like this.
They also have showcased more improvements to Gemini the assistant, which will be able to
leverage the fact that they also have search and Gmail, and can pull up some of your messages for
instance, flight or hotel info and use that to plan a trip for you. Or it can integrate with
Google sheets and look at your financials and profits get all the data together and
visualize it for you. And then you can find out that maybe bracelets is the way to go.
There are lots of things unveiled here. So when are these available? Well,
there are different dates for each of them, but most of these come in the future,
I’ve put a link in the description that tells you what is going to be available and when.
The bottom line is that after text, audio and images, now we have entered the era of AI video
understanding, there is intense competition in all of these AI areas and that only means one thing.
It means an excellent future for us consumers and Fellow Scholars. What a time to be alive!
In the meantime, I am also really excited to show this to you. I had the honor of accessing
the shiny new ChatGPT app on MacOS early, and check this out. It can be invoked with a button
combination, much like Spotlight, you just write your prompt there, and there you go. You can even
add screenshots there from a paper and ask ChatGPT about it. Mathematical typesetting
is working beautifully out of the box. Extra style points for you if you know
which paper this is from. Let me know in the comments below. And even chatting with
voice is working. It is beautiful and I absolutely love it. As far as I know,
it is going to be deployed to everyone soon, so stay tuned! And, my goodness, what a week in AI!