No, it's not those transformers.
But they can do some pretty cool things, let me show you.
So why did the banana across the road?
Because it was sick of being mashed!
Yeah, I'm not sure I quite get that one.
And that's because it was created by a computer.
I literally asked it to tell me a joke.
And this is what it came up with.
Specifically, I used a GPT-3, or a generative pre-trained transformer model.
The 3 here means that this is the third generation.
GPT-3 is an autoregressive language model
that produces text that looks like it was written by a human.
GPT-3 can write poetry, craft emails and evidently come up with its own jokes.
Off you go.
Now, while our banana joke isn't exactly funny,
it does fit the typical pattern of a joke with a set-up and a punch line and sort of, kind of makes sense.
I mean, who wouldn't cross the road to avoid getting mashed?
But look, GPT-3 is just one example of a transformer.
Something that transforms from one sequence into another.
And language translation is just a great example.
Perhaps we want to take our sentence of,
"Why did the banana cross the road?",
and we want to take that English phrase and translate it into French.
Well, transformers consist of two parts.
There is an encoder, and there is a decoder.
The encoder works on the input sequence,
and the decoder operates on the target output sequence.
Now, on the face of it, translation seems like little more than just like a basic lookup task,
so convert the "why" here in our English sentence to the French equivalent, "pourquoi".
But of course, language translation doesn't really work that way.
Things like word order and turns of phrase often mix things up.
And the way Transformers work is through sequence-to-sequence learning
where the Transformer takes a sequence of tokens,
in this case words in a sentence,
and predicts the next word in the output sequence.
It does this through iterating through encoder layers,
so the encoder generates encodings
that define which part of the input sequence are relevant to each other
and then passes these encodings to the next encoder layer.
The decoder takes all of these encodings and uses their derived context
to generate the output sequence.
Now, transformers are a form of semi-supervised learning.
By "semi-supervised", we mean that they are pre-trained in an unsupervised manner
with a large, unlabeled data set,
and then they're fine tuned through supervised training to get them to perform better.
Now, in previous videos, I've talked about other machine learning algorithms
that handle sequential input like natural language.
For example, there are recurrent neural networks, or RRNs.
What makes Transformers a little bit different
is that they do not necessarily process data in order.
Transformers use something called an attention mechanism.
And this provides context around items in the input sequence,
so rather than starting our translation with the word "why" because it's at the start of the sentence,
the Transformer attempts to identify the context that bring meaning in each word in the sequence.
And it's this attention mechanism that gives Transformers a huge leg up
over algorithms like RNN that must run in sequence.
Transformers run multiple sequences in parallel.
And this vastly speeds up training times.
So beyond translations, what can Transformers be applied to?
Well, document summaries, they're another great example.
You can like feed in a whole article as the input sequence
and then generate an output sequence
that's going to really just be a couple of sentences that summarize the main points.
Transformers can create whole new documents of their own, for example, like write a whole blog post.
And beyond just language, Transformers have done things like learn to play chess
and perform image processing that even rivals the capabilities of convolutional neural networks.
Look, Transformers are a powerful, deep learning model, and
thanks to how the attention that mechanism can be parallelized,
are getting better all the time.
And who knows?
Pretty soon, maybe they'll even be able to pull off banana jokes that are actually funny.
If you have any questions, please drop us a line below,
and if you want to see more videos like this in the future,
please like and subscribe.
Thanks for watching.