the neat thing about working in machine
learning is that every few years
somebody invents something crazy that
makes you totally reconsider what's
possible
like models that can play go or generate
hyper-realistic faces
and today the mind-blowing discovery
that's rocking everyone's world is a
type of neural network called a
transformer
transformers are models that can
translate text write poems and op-eds
and even generate computer code these
can be used in biology to solve the
protein folding problem
transformers are like this magical
machine learning hammer that seems to
make every problem into a nail if you've
heard of the trendy new ml models bert
or gpt-3 or t5 all of these models are
based on transformers
so if you want to stay hip in machine
learning and especially natural language
processing you have to know about the
transformer so in this video i'm going
to tell you about what transformers are
how they work and why they've been so
impactful let's get to it so what is a
transformer it's a type of neural
network architecture to recap neural
networks are a very effective type of
model for analyzing complicated data
types like images videos audio and text
but there are different types of neural
networks optimized for different types
of data like if you're analyzing images
you typically use a convolutional neural
network which is designed to vaguely
mimic the way that the human brain
processes vision and since around 2012
neural networks have been really good at
solving vision tasks like identifying
objects and photos but for a long time
we didn't have anything comparably good
for analyzing language whether for
translation or text summarization or
text generation
and this is a problem because language
is the primary way that humans
communicate you see until transformers
came around the way we use deep learning
to understand text was with a type of
model called a recurrent neural network
or an rnn that looks something like this
let's say you wanted to translate a
sentence from english to french
an rnn would take as input an english
sentence and process the words one at a
time and then sequentially spit out
their french counterparts the key word
here is sequential in language the order
of words matters and you can't just
shuffle them around
for example the sentence jane went
looking for trouble means something very
different in the sentence trouble went
looking for jane
so any model that's going to deal with
language has to capture word order and
recurrent neural networks do this by
looking at one word at a time
sequentially but rnns had a lot of
problems
first they never really did well at
handling large sequences of text like
long paragraphs or essays by the time
they were analyzing the end of a
paragraph they'd forget what happened in
the beginning
and even worse rnns were pretty hard to
train because they processed words
sequentially they couldn't paralyze well
which means that you couldn't just speed
them up by throwing lots of gpus at them
and when you have a model that's slow to
train you can't train it on all that
much data this is where the transformer
changed everything they were a model
developed in 2017 by researchers at
google and the university of toronto and
they were initially designed to do
translation but unlike recurrent neural
networks you could really efficiently
paralyze transformers and that meant
that with the right hardware you could
train some really big models
how big
really big
remember gpt3 that model that writes
poetry and code and has conversations
that was trained on almost 45 terabytes
of text data including like almost the
entire public web
[Music]
so if you remember anything about
transformers let it be this combine a
model that scales really well with a
huge data set and the results will
probably blow your mind so how do these
things actually work from the diagram of
the paper it should be pretty clear
or maybe not actually it's simpler than
you might think there are three main
innovations that make this model work so
well
positional encodings and attention and
specifically a type of attention called
self-attention
let's start by talking about the first
one positional encodings
let's say we're trying to translate text
from english to french positional
encodings is the idea that instead of
looking at words sequentially you take
each word in your sentence and before
you feed it into the neural network you
slap a number on it one two three
depending on what number the word is in
the sentence in other words you store
information about word order in the data
itself rather than in the structure of
the network then as you train the
network on lots of text data it learns
how to interpret those positional
encodings
in this way the neural network learns
the importance of word order from the
data
this is a high-level way to understand
positional encodings but it's an
innovation that really helped make
transformers easier to train than rnns
the next innovation in this paper is a
concept called attention which you'll
see used everywhere in machine learning
these days in fact the title of the
original transformer paper is attention
is all you need
so
the agreement on the european economic
area was signed in august 1992. did you
know that
that's the example sentence given in the
original paper and remember the original
transformer was designed for translation
now imagine trying to translate that
sentence to french one bad way to
translate text is to try to translate
each word one for one but in french some
words are flipped like in the french
translation european comes before
economic
plus french is a language that has
gendered agreement between words so the
word europe bien needs to be in the
feminine form to match with la zone
the attention mechanism is a neural
network structure that allows a text
model to look at every single word in
the original sentence when making a
decision about how to translate a word
in the output sentence in fact here's a
nice visualization from that paper that
shows what words in the input sentence
the model is attending to when it makes
predictions about a word for the output
sentence
so when the model outputs the word
european it's looking at the input words
european and economic you can think of
this diagram as a sort of heat map for
attention
and how does the model know which words
it should be attending to
it's something that's learned over time
from data
by seeing thousands of examples of
french and english sentence pairs the
model learns about gender and word order
and plurality and all of that
grammatical stuff so we talked about two
key transformer innovations positional
encoding and attention
but actually attention had been invented
before this paper the real innovation in
transformers was something called
self-attention a twist on traditional
attention
the type of attention we just talked
about had to do with aligning words in
english and french which is really
important for translation but what if
you're just trying to understand the
underlying meaning in language so that
you can build a network that can do any
number of language tasks
what's incredible about neural networks
like transformers is that as they
analyze tons of text data they begin to
build up this internal representation or
understanding of language automatically
they may learn for example that the
words programmer and software engineer
and software developer are all
synonymous and they might also naturally
learn the rules of grammar and gender
and tense and so on the better this
internal representation of language the
neural network learns the better it will
be at any language task
and it turns out that attention can be a
very effective way to get a neural
network to understand language if it's
turned on the input text itself
let me give you an example take these
two sentences server can i have the
check
versus looks like i just crashed the
server the word server here means two
very different things and i know that
because i'm looking at the context of
the surrounding words
self-attention allows a neural network
to understand a word in the context of
the words around it so when a model
processes the word server in the first
sentence it might be attending to the
word check which helps it disambiguate
from a human server versus a mental one
in the second sentence the model might
be attending to the word crashed to
determine that this server is a machine
self-attention can also help neural
networks disambiguate words recognize
parts of speech and even identify word
tense
this in a nutshell is the value of
self-attention
so to summarize transformers boil down
to positional encodings attention and
self-attention
of course this is a 10 000 foot look at
transformers but how are they actually
useful one of the most popular
transformer based models is called bert
which was invented just around the time
that i joined google in 2018.
bert was trained on a massive text
corpus and has become this sort of
general pocket knife for nlp that can be
adopted to a bunch of different tasks
like text summarization question
answering classification and finding
similar sentences
it's used in google search to help
understand search queries and it powers
a lot of google cloud's nlp tools like
google cloud automl natural language
burt also proved that you could build
very good models on unlabeled data like
text scraped from wikipedia or reddit
this is called semi-supervised learning
and it's a big trend in machine learning
right now
so if i've sold you on how cool
transformers are you might want to start
using them in your app no problem
tensorflow hub is a great place to grab
pre-trained transformer models like
burnt you can download them for free in
multiple language and drop them straight
into your app
you can also check out the popular
transformers python library built by the
company hugging face that's one of the
community's favorite ways to train and
use transformer models for more
transformer tips check out my blog post
linked below and thanks for watching
[Music]
you