hi my name is Emily Weber I'm the
machine learning specialist at Amazon
Web Services I hope you've been enjoying
our videos we've learned a lot about
notebook instances built in algorithms
using machine learning marketplace
algorithms bringing your own we learned
about training and tuning jobs now we're
gonna learn about deployment options on
Amazon sage maker and this is your deep
dive so first off there's a thing that's
called an endpoint the sage maker
endpoint what is this this is multiple
can be multiple ec2 instances in case of
multiple ec2 instances there are over
different availability zones in order to
increase your high availability when you
specify model deploy in the sage maker
Python SDK that is a single line of code
that you're gonna get to run right just
one time run that cell that's gonna spin
up a managed end point and your endpoint
that's multiple ec2 instances each ec2
instance is gonna have a webserver
that's gonna be able to respond to
requests and it's gonna have your model
artifact and so each instance is going
to be serving prediction responses those
are gonna be handled by a load balancer
so your load balancer is taking care of
health checks making sure that your
instances are up all of those are gonna
be sitting behind your model endpoint
your endpoint is a restful api stage
maker is automatically creating a
restful api for your specific model for
every model that you deploy on Sage
Maker and so this gonna operate like any
restful api you've got json accepts and
responses we see customers very commonly
using lambda to connect between your two
points you've got your api gateway when
you need information coming from the
internet to hit your AWS services and so
most commonly you're going to be
securing your services with a V PC so a
virtual private cloud that's making sure
that your resources are secure you're
gonna open up access between that cloud
and the Internet using the API gateway
and then AWS lambda can be a reference
point you're not locked into lambda you
can actually go straight from your
endpoint to the API gateway or you can
use what's called an inference pipeline
which we're going to learn about in a
second here to actually set that up
Landa just gives you lots of flexibility
and customers love it so that is your
sage maker end point so your sage maker
end point is best for cases of online
inferencing when you have data that's
coming in from the internet and you need
to serve prediction responses in real
time so we're talking sub-second latency
right you need your responses to be fast
and that's what your sage make your
endpoint is gonna be for let's check out
an example so over here this training
job was from a one that we ran
previously so this was for a blazing
text so essentially the scenario here
right we've got those wikipedia articles
and those wikipedia articles are labeled
based on the content of that article and
then we plug those into the sage maker
built an algorithm called blazing text
and just to recap here so that happened
down here right that's our container
that's our role that's our resource
utilization and then we specified our
channels we've got our train of
validation channel we set those up down
here then we called model dot fit and in
this case we got all the logs printed
out down here
we're gonna hit model dot deploy and so
this is another cluster right it's
another ec2 instance this was is an m4
Excel and then we're gonna reference
that classifier right here and so when
we want prediction responses those are
actually gonna go up and hit our
endpoint let me show you so in the
console we can cruise down and look at
endpoints and so this is our blazing
text endpoint it's status is up this is
the URL so that's actually the one that
you're gonna hit when you're making
calls and here's our invocation metrics
right so we've got we've got these nice
charts here they're gonna tell us how
its performing that's the name of our
model our model is gonna have a
production variant that production
variant tells us what percentage of
traffic is going to be hitting that
model for every point in time and so
over here the two examples that we're
sending up one is about convair and
another is Berwick Secondary College and
both of those come back with different
classification responses right so the
first one has a probability of 99
percent and that label is a company the
second one down here is a probability of
also over 99% and the label is an
educational institution let's have some
fun right so I'm gonna I'm gonna delete
these sentences and I'm gonna paste in
I'm gonna say all right there we go
so that is hitting our end point right
that's actually um being packaged up
it's a RESTful API so we have to
actually dump it into JSON then we just
called text classifier dot predict right
that hits the end point we get the
response back which is another JSON
object so we have to decode it and then
we can actually look at the prediction
responses so some clothes some pro tips
here just to close things out right
definitely turn your endpoints off when
you're when you're not actually using
them you can write lambda functions to
do this definitely leverage the
servantless architecture that you have
in order to turn your instances off when
you're not actually using them you can
also use what are called inference
pipelines to manage both pre and
post-processing so that's where you can
create a number of containers up to five
containers and then actually just run
them sequence to sequence so you'll have
in a one container that's managing your
feature pre-processing that's gonna pass
it into another container where you're
doing your model inferencing which will
then pass it out to a post processing
and that's gonna be your if-then-else
statement based on the confidence level
that your model actually identified so
you can deploy that entire inference
pipeline as a single endpoint those
containers are going to be co-located on
the same ec2 instance so just make sure
you're picking your resource correctly
so that you're not gonna run into any
memory or disk errors you can also have
more models in a single docker container
right so even though a docker can it's
one docker container per training
cluster you can definitely have multiple
models in that container you just have
to write the code you know
in a way that in the way that does the
job and also making sure that you have
sufficient infrastructure to get that
done
I definitely think about the amount of
data that's hitting a single end point
some of the limitations there are such
that you want to not be sending too much
data for an end point if you've got you
know more than a handful of megabytes
and definitely you want to be sending
that up against the batch transform
rather than the end point um and just to
hit this home you can absolutely train
your model somewhere else and then host
it in Sage Maker right so you're not
locked in on doing this step by step if
you like your training platform on your
DAP on your laptop or you like your
training platform somewhere else look at
some of the information on our blog
series about getting that into the right
format and then actually just hosting it
in Sage maker so you can use the
endpoints or the batch transforms and so
thank you that's all got my name is
Emily Webber I'm a machine learning
specialist at Amazon Web Services we
just learned about deployment options
and if you're interested definitely
check out our github sites thank you
very much
you