so over the last couple of lectures
artemy has been telling you about
neural networks and how how many how
these major advances have been coming
about as a result of this new technology
in this in this lesson we're more going
to talk about the caveats of machine
learning and things to think about
before you deploy these new systems so
one of the major piece of advice that we
have for you is that data and questions
come before algorithms so that is a lot
of machine learning problems don't need
advanced techniques like deep neural
networks with LST M's and rel use and
all these fill these other buzzers right
a lot of questions can be solved by
having a really clear question that
you're answering and a really good data
set that you have to answer that
question and so the first things to
think about are what what do I want to
know and if I can't know what I want to
know is there something that will be
useful anyway and secondly do I have the
data to answer that question that if I
don't have that data how can I go about
and get it and so neural networks
oftentimes really need really large data
sets in order to work properly and for
smaller data sets there are other
different algorithms that work better
with small data and so this is one thing
to think about is trying these other
algorithms before jumping on the neural
network high train and again to
emphasize think about data and questions
first before we think about algorithms
so on that point about clean data though
another point we want to make is that
it's not always obvious how to make data
clean and so in particular certain
things are obvious right you can
represent age as a number you can
represent zip code as a number but other
things aren't so obvious so let's
imagine we want to build a product
recommendation system and you've seen
that someone in the past has built has
bought a tripod they've bought a
microphone well how do we take that data
and make new recommendations right one
way you could do it is to literally list
the product number that they bought but
that might be really confusing maybe you
want to represent that as they bought a
tripod so maybe if you see they've
bought a tripod and they bought a
microphone they may also need a camera
but the same time if you see that
they've been looking at six different
kinds of tripod maybe you do want those
as exactly the tripod that they are so
you can make a different recommend
for another tripod for instance so
another thing is that natural language
data can be really difficult to
represent and so you might take each
individual word and make it a number but
a lot of times these words combine in
difficult ways now hypothetically you
could take an entire sentence and
represent it as a number but that's not
going to be very useful to use because
most sentences aren't used all that many
times so again it's really tricky and
requires a lot of research to figure out
how it's a map something like natural
language data and have it say oh this
person likes me or doesn't like me so
the other caveat we want to mention is
that is this concept of hardware so
these neural networks and other machine
learning algorithms can be oftentimes
need to be trained on large data sets
and they need to run through this data a
lot of times and so the thing is is that
it often takes a lot of hardware in
order to practically train these models
and so alphago which we talked about a
couple of lessons ago that actually ran
on something called a tensor processing
unit which is a specially designed piece
of hardware in order to run fast and so
you don't necessarily need that much
customized specialized Hardware but it
is the case that oftentimes you need
lots of graphics cards or access to
something like the Amazon Cloud in order
to practically train these algorithms so
it's another thing to think keep in mind
of depending on how long you have to
train there can be different trade-offs
and machine learning algorithms a third
thing though to talk about which i think
is big is this idea of transparency and
robustness so many of you may have seen
projectile motion either an introductory
physics class and let's think about we
have paratroopers that are going to jump
out of an airplane and we're gonna take
Newton's Newton's laws of motion with
projectiles that say well the velocity
with which they hit the ground is a
square root of two times the height they
jumped times the acceleration due to
gravity and if we run those numbers we
find that they're gonna hit the ground
at a hundred and twenty miles an hour
that's a really fast speed so they
probably shouldn't be jumping right
wellnot right so we know Newton's laws
and with projectiles ignore air
resistance and these paratroopers have
parachutes which and operate entirely
under air resistance so we know that
this
is going to be wrong because of air
resistance being around ok so here's
another example there's a neural network
that classified this Pando with 50 7.7%
and it classified this very close image
of a panda as a given with 99.3% so why
is it that the neural network made this
mistake well it's really not clear right
with and so this is one of the major
benefits to say scientific discoveries
over pure machine learning in science we
often know what the assumptions are and
when those assumptions don't no longer
hold well with machine learning
algorithms and particular the more
advanced machine learning algorithms we
don't really know what core assumptions
are operating under and therefore when
we would expect them to fail so this is
something to keep in mind before you
deploy or use a machine learning
algorithm is to understand what you need
to know in terms of what kinds of
mistakes that it's going to make or not
in this particular example was made with
a very particular kind of thing called
an adversarial input where they knew how
the neural network behaved and they took
that original Panda image and found a
very small change in the image that's an
imperceptible change to humans but to
the neural network made a very big
difference and so there are kind of two
messages to come here one of which is
that while it seems like machine
learning algorithms are intelligent and
often reproducing human intelligence
it's certainly sure that they don't make
mistakes and the same way humans make
and the second thing to be a wary of is
this kind of security issue in that
there are these kinds of inputs that can
completely fool the system in ways that
are really unclear and so it's important
to keep these kinds of things in mind
before you just let them loose and let
them do whatever they want another major
caveat we want to talk about or what
we're calling feedback effects that is
when the neural network influences its
own predictions and so a simple example
of this is that t caste system that was
mentioned way back in Lesson one where
it was this sort of series of if
statements that told a pilot how to
behave in order to try to avoid two
pilots colliding one problem with the
original T caste system is it didn't
know about something called the ground
and so it oftentimes make bad
predictions to pilots say when they were
taking off and landing and so the system
wasn't really sophisticated enough to
deal with the full environment in which
it was deployed and so part of the
problem was that pilots now sort of
learned to not really trust the system
because it wasn't making the decisions
that they hoped it to make and so
because of this feedback loop the system
wasn't as effective as they hoped it
would be originally and had to upgrade
the system several times so let's
consider another example from
epidemiology where maybe we're gonna
model a disease outbreak and we're gonna
try to see how quickly it's gonna spread
through the population now these models
often assume that I'm gonna do what I
normally do so I'm gonna go to work I'm
gonna come home maybe I'll go to a
friend's house and that's gonna
influence who I interact with and
therefore who I can spread the disease
to however if I'm going to take this
epidemiology model and say oh my gosh
this is really bad and then announce to
everybody that we should be really
careful well now all the sudden maybe
I'm not going to be going to work maybe
I'm going to do something entirely
different instead and maybe in fact I
have the disease but I'm going to go to
an entirely different City to avoid it
thus spreading the disease much faster
than if I hadn't used the model so it's
important to make sure that the model
kind of isn't influencing how the system
that it's modeling is going to behave
and another kind of example of this
comes from something called recidivism
in parole decisions so deciding if you
should grant parole or not to
individuals and there are some issues
with fairness that are being researched
now in terms of how fair or say biased
are these decision-making algorithms and
that's one major thing to think about
and the second related thing to think
about is let's say we don't grant parole
to a certain class of individuals like
our algorithm just says never never
grant them parole well we're also if we
want to collect data in the future we're
never getting data on that category of
individuals and so we never learned that
our system maybe maybe was wrong and so
again if the machine learning algorithm
is influencing the data that then is
going to train the next iteration of the
machine learning algorithm you really
need to be careful so another thing to
think about is that often times problems
aren't even AI problems at all so there
are lots of
that involve humans getting together to
make decisions as collectives for
example political problems or to deal
with emotional or relationship issues
like social problems and in many cases
just computers aren't there to solve
these problems for us these are human
human problems that involve conflict
resolution and other very important
discussions that we just can't program a
computer to tell us the right solution
another example though is that there are
learning problems that aren't machine
learning problems so as mentioned
machine learning is generating new
predictions from data but a lot of
problems don't require data they require
insight and so in the middle here for
example is there's a text that was looks
like gibberish but in fact it was
generated according to something that's
called a Caesar cipher and so what a
Caesar cipher is is that for each letter
in the alphabet we replace it with a
different letter in the alphabet so for
example a becomes F and B becomes Q etc
and so if you have insight what you can
realize is that in English texts the
letter E occurs the most often so maybe
we can count up the letter that occurs
most often and make that in E and then
by doing this procedure you can
translate that back into its original
text but we see here right that this
required two levels of insight it
required that we understand that this
text isn't gibberish maybe it does
represent something real and the second
thing was how to go about this
translation process it didn't require
lots of data
it required insight to see that that's
there another example of this is the
code talkers during world war ii where
instead of taking english encrypting it
and sending it over the radio instead
they had native navajo speakers speaking
over the radio so if you're German and
have never heard Navajo before this
sounds like gibberish and maybe you
think that it's this text is somehow
encrypted well instead it's a different
language entirely and if you had the
insight to realize that it's Navajo
maybe you could do something about it
but in Germany there weren't very many
native Navajo speakers and so they never
noticed this so to summarise machine
learning is good and has made lots of
major advances on problems that
previously had been hard however there
are these really important caveats to
consider and so our advice
you is to have clean data and clear
questions before you start to think
about algorithms make sure you have what
you need to answer the question before
you think about ways of how to answer
the question and so and also then before
you deploy this system we encourage you
to think about all these caveats that
we've mentioned for example transparency
and different kinds of feedbacks that
the system might have with itself