TITLE: [MUSIC]
PHIL MAZENETT: Welcome to today's talk.
My name's Phil Mazenett. I work at OctoML.
At OctoML, we work with
customers to get models into production.
In today's talk, what I want to talk to
you guys about is, with
all that information from customers that
we see about models going into production,
what are some of the mistakes that we see
that people continue to go through
and how you can avoid those mistakes
so you can get your models to
work much faster and actually generate some revenue?
Generally, what we see when we're
talking to customers is that
customers are trying to get the models
into production in whichever way they can.
They're brute forcing these deployments
by passing a lot of
different frameworks or creating specific,
very rigid pipelines when
it comes to getting those models into production.
This, of course, creates a lot of
mess when you have to introduce a new model,
when you have to go in and work with
different new frameworks or anything that
basically goes outside of what was already created.
Generally, what also gets caused as
a result of this is folks are basically
passing on repos that contain
the environments that they have in
their current scenarios in
the ways that they're training these models.
When that happens, a lot of the nuanced information that
needs to be known about how the model gets
deployed gets lost in translation,
and it becomes very
difficult for the folks that are actually
deploying these models to do it in a very efficient way.
That, of course, causes project failures and delays,
which is one of the main reasons that a lot of
these models don't make it into
production fairly quickly.
That, as a result, also causes a number of folks to then
have time in their hands in inappropriate ways.
By that, meaning,
people are essentially going out there and
doing a lot of trial and error
when it comes to rebuilding
the environments that are necessary to actually build
the models and deploy them into production.
Then, for anybody else that is downstream from that,
it causes a lot of sideline or a lot of waiting and
basically, misuse of the time
that you could be doing to deploy further models.
This brings us to a point that, if you look at some of
the information that VentureBeat put out in 2019,
that 87 percent of the models that are actually
trained and that folks are working
on actually never make it to production.
If you actually think about it, that means 13 percent of
the models that are actually
being worked on make it to production,
which is actually a really low number.
If you think about that and
think about that each of those models takes
about a quarter of a year to actually get to production,
it means that there's a tremendous amount of waste
that's actually happening, because no matter
what, models, before they get to
deployment, still have to be trained,
still have to consume some data.
There still has to be a lot of
investment people-wise and a number of
different other things that go into building a model.
If we really take a second to stop
thinking about what this actually means,
it means that we're actually generating a lot of
work that is not returning any value for enterprises.
Let's talk about some of the mistakes that we've seen
customers do when it comes to actually deploying models.
There is definitely a lot of
learnings that we've gotten from talking to
a bunch of different customers
in our company.
Really the number one mistake that we've
seen is around not taking
into account the end-state hardware target
where the model is going.
What I mean by that is, models are typically
trained in environments that are very large,
that are distributed,
that enable the use of Clouds, for example,
like the Azure ML Studio,
and the use of GPUs, like an Nvidia GPU,
and then not taking into account that
this needs to go into a bunch of different environments;
sometimes those environments cannot be selected.
For example, you may have to deploy in Intel CPUs or
AMD CPUs, versus the Nvidia GPU
that you thought that you were building the model for.
There's also a component where
somebody has to figure out, what
is really the most cost-effective
way to deploy this model?
Sometimes that decision, actually
alone, dictates where the model is going to go,
and this is not even incorporating
something, for example, like, if I need to
deploy the model into an edge device,
what happens with that particular state?
The edge device may actually not be able to run
the model at all because it
doesn't have the necessary power.
This is a problem that we see
people usually having to deal
with quite a bit, and some of
the solutions that we've seen
customers take are doing things,
for example, like exploring quantization, or pruning, or
other techniques that, of course, change
data types to be able to achieve a smaller model,
something that is much easier to consume for a lot of
devices at a cheaper cost
in terms of the actual running.
However, that, of course,
means that some of the accuracy
of the model can go away.
If you're changing data types,
and you're changing the precision of the model,
that could mean that the model is no longer
valid, so there's obviously
something to think about there.
Another way to solve this problem that we've seen from
customers is really just throwing
more compute at the problem.
That's obviously a really good solution
up until a point, of course,
because if you think about it, as more compute exists,
people are building more complex models.
When people build more complex models,
they require more compute and the circle
continues to go on, and essentially gets to
a point where it's not even functional or
useful to be able to run these models at
any point in time, because they're just so costly to run.
We're starting to see that with some of
the larger natural language models
where they're just getting really big.
These are models that are not going to
run outside of data centers.
The last thing you can do,
or we've seen customers actually do,
is optimizing the model in a different way.
So, not using things like quantization and pruning,
but actually looking to see how you can use
the stack of the machines that the model is running at.
What I mean by stack, I mean software stack.
So, for example, using the CUDA libraries appropriately,
using the Intel DNN libraries,
or using even better ways to compile
the model for the instruction set
of the particular hardware target.
There are companies that can do that for you, in terms
of an actual automated process,
and there are solutions that are
used by those companies, for example,
TVM was a project that was created from
Open Source to be able
to approach this particular problem,
which is essentially replacing the tuning
of the model that is done in a hand
tuning way by using machine learning for it.
OctoML, it just happens
to be a company, which is the company I
work at, which utilizes TVM as the solution for that.
Now, there are other solutions you can use.
So, you can use other technologies,
like Tensor or TN ONNX Runtime
that do similar approaches.
Now that we covered that, there's
another problem that we keep seeing with our customers.
It's a little bit different than
the problem that we just talked about a second ago.
This problem is less related to the cost and
the performance that you have to
think about when you're actually running the model.
It's more related to
the options that you have
when you're thinking about deploying the model.
What I mean by that is,
obviously, in the training space,
there's a lot of room for possibilities.
Folks have access to things like Hugging Face,
and all these different libraries
that are happening that are coming out,
that make model creation much simpler.
You're able to basically leverage the knowledge of
many people that have
built these different frameworks for you.
The problem with those is that
those particular frameworks have
requirements that are very specific to the type
of machines that they've been compiled for.
Of course, if you're taking
a model that has been created with PyTorch with
some new library, and you want to put
that into a Kubernetes cluster,
say that has Nvidia GPU and
one that has an Intel CPU, then you're probably fine.
However, let's say that
your actual product needs to
run on a phone or on an edge device,
and those different devices have versions of an OS or
have different types of chips that are not
supported by Docker approach,
then now you have to start thinking about, how
do I generate assets from
the original model that can run
in an efficient way in that particular device?
So, some of the things that
people are doing is, of course,
doing conversions to things like
TF Lite, which is what you see here.
But, if you take a look very closely in the diagram,
that means that you essentially have to create
a completely different approach to deploying this model.
You basically have to maintain
different pipelines, and that,
of course, causes a ton of stress on the organization.
Now, there's two ways that we've
seen customers address this particular issue.
The first way, which essentially I just talked about,
which is customers are
basically going in and saying, okay,
we have a number of different frameworks and
different devices that are
important for us as an organization,
so we'll go ahead and build pipelines around those.
We'll put some guardrails and limitations,
but it'll allow us to do an automated, ongoing process.
Now, the problem with that, of course,
is that if you do that,
you're essentially limited by whatever
the pipelines can do for you.
The minute that you have to use
a new framework, or that something
changes within the framework, or
something changes within the device,
you have to start creating a new
pipeline or a new workflow.
Most likely, you can't get rid of
the old one, and you have to basically
maintain a new one and continue to
add these number of pipelines that you have to support.
Another option, which is
a much better option for this type of
problem, is to essentially look at how you
can recompile the model for the target
and for the actual stack of
the target where the model is going to run.
What I mean by that is, instead of thinking,
how do I package
all the different dependencies
that I need to run this model,
let's actually convert this model into
something more universal that can actually
run in all these different devices
and take advantage of the devices
that are the instructions and
the accelerations that the devices provide.
There are ways to do it. The way that we are tackling
mostly in our company is using a container approach.
There's also ways that you can do it
with shipping C libraries, and so on.
That being said, this allows the organization to
have a much more fluid way to look at
deployment, and basically have
a translation layer that allows you to always develop
models, in whatever frameworks you
have, without worrying about
how they need to run on the end state.
The third mistake that we see is actually more of
a combination of everything that we've seen.
It's really, I would say,
the most impactful problem that we see in deployment.
That is, this concept of there is a wall,
and there's something that you have to throw,
as a machine learning researcher,
your model that you've built and
everything around that model into,
let's call it, a Git repo in a Gerrit ticket.
Then, assume that somebody will be
able to pick that up and actually rebuild your model.
That is very common in
many workplaces and many enterprises.
The problem with that is
that, essentially, you're putting a lot
of work on the folks that are going to deploy the model,
whether they're infrastructure or some other team,
to actually go in there and understand what it
is that you did, as an ML researcher,
when it comes to building this model and getting
the model to be as accurate as you want it to be.
Some of the things that
those folks have to do are things like,
for example, building the environment,
making sure the right Python packages are
there to actually run
the code the way it's supposed to be running.
Other things you may have to do is clean up your code,
so make sure that the code is
efficient and it's not crashing
and it's not running out of memory, or things like that.
I mentioned this before, if we really think
about mission- critical workloads,
some of those folks may actually have
to take a look back at
that entirety of the code, and think
about how they translate it to a better language.
For example, get away from Python and
use something like C.
C is a much more dependable language.
It's something that you'll probably want to run, and
something that needs to make
sure that it's mission critical.
But the problem with that, of course,
means that these people that are
actually doing that
recompilation or translation to C
have to fully understand what's
actually happening with the model, and have
to fully understand the mathematical
properties of the model, so
that they can translate it to
that particular piece of C code.
On top of that, those people actually
have to understand the stacks that are
necessary to run in
those particular machines and
how you can get those accelerated.
For example, they need to be CUDA experts,
or they need to be VNNI experts,
or whatever is the library of interest.
Of course, that's a very
complicated set of folks to find.
So, if we really think about it, those are
huge problems that, essentially,
the research team is putting on
the infrastructure team by
just throwing the model over the wall.
The way that we like
to think about it, and we've been talking to
customers about how to fix this, is
that you essentially have to
remove this concept of the wall
out of the equation and think of models as
something that is much closer
to functions in a software program,
essentially, something that you train
to make some decisions based on some data,
which we all know what that models are,
and then treat them as a function,
like any other piece of software.
By doing that, you can essentially
start inserting yourself earlier
into the process of
the DevOps or software development process.
That way, you can essentially start
informing some of the folks that are
working alongside you on,
what are the things they need to be thinking about.
For example, with software developers,
you can provide them quicker access to
a prototype of the model that you're actually working on.
They'll be able to actually work
with that model, test something,
even if it's an early version of the model,
and be able to actually start
building the app the right way.
From the infrastructure point of view,
these folks are also going to be able to
understand, what are
the resource requirements of this model,
so that they can very
quickly advise if the model is or is not
going to be able to run in the targets or
the hardware that you intend
to run the actual model.
For example, this would solve something, like,
your model doesn't run on
a Raspberry Pi, or whatever is of interest.
If we do this in a much more iterative way,
in a much more connected way,
that would essentially enable us to
really take advantage of all the stuff that we
learned when it comes to
Agile software development and
bring that over to the model space.
By doing so, we can essentially get
models faster out the door,
we can break down this whole concept of
only 13 percent of the models
are making it to production,
and we're able to get a much better cycle of
reproducibility and iteration when it comes
to rebuilding the next version of the model, and so on.
I hope that was a helpful point to
start talking about ways that we can really
get models into production faster
and what else we can do to avoid these mistakes.
Thank you so much for taking the time
to go through this presentation with me.
If you do have any questions,
please feel free to reach out to me
at any point in time.
TITLE: [MUSIC]