if we're moving from cat to dog which is
similar things so we go away from cat
and towards dog
and then we go can i go beyond in that
direction yes so the first result is
dogs which is kind of a nonsense result
the second is pit bull
so that's like the doggiest of dogs
right the least cat-like dog that feels
right yeah yeah actually well if you go
the other way well the the most cat-like
cat
the most undog-like let's find out it's
gonna be kitten right it's gotta be
cats feline kitten it's not really
giving us anything much to work with
i thought i would talk a little bit
about word embeddings word devec and
just wording bettings in general the way
i was introduced to word embeddings or
the the sort of context that i'm most
familiar with them in is like
how do you represent a word to a neural
network
well it's a set of characters isn't it i
mean
need it be more than the set of
characters that make it up
right so you can do that but you
remember the thing we were talking about
before in language models you have a
problem of how far back you can look i
would much rather be able to look back
50 words than 50 characters
um
and like if you if you're training a
character-based model a lot of the
capacity of your network is going to be
used up just learning
what characters count as valid words
right what combinations of characters
are words and
so if you're trying to learn something
more complicated than that you're
spending a lot of your time training
just like what words are and a lot of
your network capacity is being used for
that as well
but this isn't a hard problem we know
what the words are right you can give
the thing a dictionary and then you're
kind of
that gives it it gives it a jump start
the point is neural networks
they view things as like a vector of
real numbers or a vector of floats which
is like
some of the real numbers um
and so if you think about something like
uh an image
representing an image in this way is
like fairly straightforward you just
take all of the pixels and put them in a
long row and if they're black then it's
zero and if they're white then it's one
and you just have grayscale in between
for example it's like fairly
straightforward and so then you end up
with a vector that represents that image
it's a reasonably good representation it
sort of reflects some elements of the
structure of what you're actually
talking about
so
um
like if you take if you take the same
the same image and make it a little bit
brighter for example
that is just making that vector a bit
longer right or a point in that
configuration space that's a bit further
from the origin you can make it darker
by moving it close to the origin by
reducing the length of that vector if
you take an image and you apply a small
amount of you know noise to it
that represents just like jiggling that
vector around slightly in that
configuration space so you've got
you've got a sense in which
two
vectors that are close to each other
are actually kind of similar images
and um that
some of the sort of directions in the
vector space are actually meaningful in
terms of something that would make sense
for images and the same is true with
numbers and whatever else and this is
very useful when you're training because
it allows you to say if your neural
network is trying to predict a number
and the the value you're looking for is
10 and it gives you nine
you can say
no but that's close
and if it gave you 7000 you can be like
no and it's not close
and that's gives more information that
allows the system to learn and in the
same way you can say yeah that's almost
the image that i want
um
whereas
if you give the thing a dictionary of
words
say you've got your 10 000 words and the
usual way of representing this is with a
one heart vector
if you have ten thousand words you have
a vector that's ten thousand
long ten thousand dimensions and all of
the values are zero apart from one of
them which is one so like the first word
in the dictionary if it's like a
then
that's represented by a one and then the
rest of the ten thousands is zeros and
then the second word is like a zero and
then a one and then all zeros and so on
but there you're not giving any of those
clues if the thing is looking for one
word and it gets a different word
all you can say is yeah that's the
correct one or no that's not the correct
one something that you might try but you
shouldn't because it's a stupid idea is
rather than rather than giving it as a
one-hot vector
you could just give it as a number
but then you've got this indication that
like two words that are next to each
other in the dictionary
are similar
and that's not really true
right like if you have a language model
and you're trying to predict the next
word
and it's saying
i love playing with my pet
blank
and like the word you're looking for is
cat and the word it gives you is car
lexicographically they're pretty similar
but you don't want to be saying to your
network uh you know close that was very
nearly right because it's not very
nearly right it's a nonsense prediction
but then
if it said like dog
you should be able to say
no but that's close
right because that is a plausible
completion for that sentence and the
reason that that makes sense is that cat
and dog are like similar words what does
it mean for a word to be similar to
another word
and so the assumption that word
embeddings use is that two words are
similar if they are often used in
similar contexts
so
if you look at all of the instances of
the word cat in a giant database uh you
know a giant corpus of text and all of
the instances of the word dog
they're going to be surrounded by
you know words like pet
and words like you know feed and words
like play and you know that kind of
thing cute etc right
and so that gives some indication that
these are these are similar words the
challenge that word embeddings are
trying to come up with is like how do
you represent words as vectors
such that
two similar
vectors
are two similar words
and possibly so that directions have
some meaning as well
um
because then that should allow our
networks to
be able to understand
better what we're talking about
uh in in in text so the thing people
realized was if you have a language
model that's able to get good
performance of like predicting the next
word in a sentence
and the architecture of that model
is such that it doesn't have that many
neurons in its hidden layers
it has to be
compressing that information down
efficiently so you've got
the inputs to your network let's say for
the sake of simplicity your language
model is just taking a word and trying
to guess the next word so we only have
to deal with having one word in our
input but so our input is this very tall
thing right ten thousand tall
and these then feed into
a hidden layer which is much smaller i
mean it's more than five but it might be
like
a few hundred maybe let's say 300 and
these
are sort of the connections all of these
is connected to all of these and it
feeds in and then coming out the other
end you're back out to 10 000 again
right because your output is
it's going to make one of these high you
do something like soft max to
turn that into a probability
distribution
so you give it a word from your
dictionary
it then does something
and what comes out the other end is
probability distribution where you can
just like look at the highest value on
the output and that's what it thinks the
next word will be and the higher that
value is the more like confident it is
but the point is
you're going from 10 000 to 300 and back
out to 10 000. so
this 300 has to be
if this if this is doing well at its
task this 300 has to be encoding sort of
compressing
information about the word
because the information is passing
through
and it's it's going through this thing
that's only 300 wide so in order in
order to be good at this task
it has to be doing this so then they
were thinking well how do we pull that
knowledge out it's kind of like an egg
drop competition
is this why you have to devise some
method of
safely getting the egg to the floor
right it's not like the teachers
actually want
to get an egg safely to the ground right
but they've chosen the task such that if
you can do well at this task you have to
have learned some things about physics
and things about engineering and
probably teamwork yeah right right
exactly
so it's the it's the friends you make
along the way
so um
so the way that they the way that they
build this is
rather than
um
trying to predict the next word although
that will work that will actually give
you
word embeddings but they're not that
good because they're only based on the
immediately adjacent word
you um
you look sort of around the word so you
you give it a word
and then you sample from the the
neighborhood of that word randomly
another word
and you train the network to predict
that so the idea is that at the end
um
when this thing is fully trained
you give it any word and it's going to
give you a probability distribution over
all of the words in your dictionary
which is like how likely are each of
these words to show up
within five words
of this first word
or within 10 or you know something like
that if
the system can get really good at this
task
then the weights of this hidden layer in
the middle have to encode something
meaningful about that input word
and so
if you imagine the word
cat
comes in
in order to do well the probability
distribution of surrounding words
is going to end up looking pretty
similar to the output that you would
want for the word dog
so it's going to have to
put those two words close together if it
wants to do well at this task
and that's literally all you do
so so so
if you run this
on a lot it's it's absurdly simple right
but if you run it on a large enough data
set and give it enough compute to
actually
perform really well
um
it ends up
giving you each uh giving you for each
word
uh a vector
that's of length however many
uh units you have in your hidden layer
which
for which
the the nearbyness of those vectors
expresses something meaningful about how
similar the contexts are that those
words appear in
and our assumption is that words that
appear in similar contexts are similar
words
and uh
it's slightly surprising how well that
works
um and how much information it's able to
extract so
it ends up being a little bit similar
actually to the way that the
generative adversarial network
uh does things where what we're training
it to produce good images from random
noise and in the process of doing that
it creates this mapping from the latent
space to images by doing
basic
arithmetic like just adding and
subtracting vectors
on the latent space would actually
produce meaningful changes in the image
so what you end up with is is that same
principle but for words
so if you take for example the vector
and it's required by law that all
explanations of word embeddings use the
same example to start with so
uh if you take the vector for
um king
subtract the vector for man
and add the vector for woman you get
another vector out
and if you find the nearest point in
your
word embeddings to that vector it's the
word queen
and so there's a whole
giant swathe of like
ways that
ways that ideas about gender are encoded
in the language which are all kind of
captured by this vector
which we won't get into but it's
interesting to explore
i have it running and we can play around
with some of these vectors and see where
they end up so i have this running in in
google collab which is very handy i'm
using
word embeddings that were found with the
word to vec algorithm using google news
each word is mapped to 300 numbers
let's
check
whether
what we've got
satisfies our
first condition we want
dog and cat to be relatively close to
each other
and we want cat to be like further away
from car
than it is from doc right we can just
measure the distance between these
different vectors i believe you just do
model.distance distance between car and
cat okay 0.1
and then the distance between let's say
dog and cat 0.23
right
dog and cat are closer to each other
this is a good start right
and in fact we can
uh
let's find all of the words that are
closest to cat for example
okay so the most similar word to cat is
cats
makes sense followed by dog kitten
feline beagle puppy pup pet felines and
chihuahua
right so this is already useful it's
really handy that you can throw any word
at this and it'll give you a list of the
words that are similar
whereas like if i put in car i get
vehicle cars suv minivan truck right
so this is working the question of
directions
is pretty interesting
so yeah let's do the classic example
which is this if you take the vector for
king subtract the vector for man add the
vector for woman
what you get
somewhat predictably is queen and if you
put in
boy here
you get girl if you put in father
you get mother yeah and it if you put in
shirt you get blouse so this is
reflecting something about gender that's
that's in the in the data set that it's
using this reminds me a little bit of
the unicorn thing where
it you know the transformer was able to
infer all sorts of or appear to have
knowledge about the world because of
language right right but the um
the thing that that i like about this
that that
is that
that transformer is working with
uh 1.5 billion parameters and here we're
literally just taking each word and
giving 300 numbers
you know if i go
from london
and then subtract uh england
and then add um
i don't know japan
we'd hope for tokyo we'd hope for tokyo
and we get tokyo
we get tokyo twice weirdly tokyo tokyo
why is oh oh sorry it's no we don't we
get tokyo and toyco
ah which is a typo i guess and so yeah
uh usa
in new york ah okay interesting maybe
it's thinking larger city of yeah right
right like the exact relationship here
isn't clear we haven't specified that
what does it give us for australia
i bet it's yeah it's sydney sydney
melbourne so it's yeah it's not doing
capital
it's just the largest city right
um
but that's cool
it's cool that we can extract the
largest city and like this is completely
unsupervised
it was just given a huge number of
news articles i suppose and it's pulled
out
that there's this relationship and that
you can follow it for different things
you can take the vector from pig to oink
right okay and then like you put cow in
there
that's mu
you put a cat in there and you get
meowing you put dog in there
you get box
right close enough for me yeah yeah you
put um but then then it's it gets
surreal you put santa in there
ho ho
right fantastic
what does the fox say
it says phoebe
what so it doesn't know basically
although the second thing is chittering
do fox's chitter
i don't know
not in this data set