- Say you're a ride-sharing company
and you wanna create an ML model to get an idea
of how many people you'll be serving on a given day.
You start collecting data on January 1st.
You pull 100 days worth of data
and train your model on that.
After 300 days, you triple your data set,
and then you make a prediction for December 31st.
That should be your most optimized model.
More data always means a better model, right?
Not necessarily.
When it comes to training machine learning models,
it doesn't matter that you've tripled your data
if you missed the context of seasonal usage patterns.
It would be like trying to predict
how much snow you're gonna have in the winter
by looking at how much snow fell in the summer.
Model Drift, which is often referred to as model decay,
is the gradual loss of a model's prediction power due
to changes in context.
Data can and will change over time.
The accuracy of a machine learning model is at its best
when the training data matches real-world data.
The model deteriorates
as the world it is trained to predict, changes.
Now, think about operating ML models at the edge.
When you have a large amount of anything,
it's hard to keep track of everything.
With edge devices,
you may have a model deployed thousands of times.
You have to update and redeploy thousands of instances
and things quickly become complicated.
We have software pipelines to operate at scale.
ML systems are software systems,
and to realize their full potential, we need ML pipelines.
To talk about the implications of model drift
and operating machine learning at scale,
let's talk with Kavitha Prasad from Intel.
Hey, Kavitha!
Thanks for chatting with me today.
- Hey, Chris, it's great to be here
and thanks for this opportunity, appreciate it.
- Well, I've been thinking
about the way we train machine learning models.
And when we talk about the challenge of model drift,
I've learned a lot about how we improve software
with thinking about the changes in source code,
the build process and the deployment process with DevOps.
What benefits are we getting
from applying life cycle management like that
to machine learning?
- So, the MLOps is in a lot of ways similar
to DevOps Chris, but there are few changes
that needs to be kept in mind, right?
Like besides just the code versioning that you talked about,
you need a place to save the data
and also save the model versions
because that is different from DevOps.
The other key thing is, your software code
does not degrade like your machine learning models degrade.
So you have to keep that into account,
which is again, slightly different from DevOps.
So by deploying these MLOps,
there are a lot of advantages, right?
Like, you can have the data scientist focus more
on model development, while the IT infrastructure,
or the IT resources, can focus on going and deploying this
because this is a fixed, this is an MLOps pipeline.
And quality of predictions also improve
because now you have,
they've brought in the concepts
of retraining into this entire system.
So your quality of predictions improve.
And last but not the least,
the customer ease of use is very, very critical
so that this MLOps makes the development to deployment
much easier and also takes care
of the data and model validation
and evaluation of its performance in production,
and where you have deployed.
So you get meaningful insights,
so there's a lot of advantages to these MLOps being deployed
in the AI use cases.
- That makes sense, and as you were saying that
I was thinking, well I still see some connections,
for example data as source models as what you deploy.
And I feel like this becomes really important
as we scale out and start thinking about deployment
of machine learning models to the edge,
where inference is happening.
So maybe, do you see more ML use cases moving
to the edge or is that just in Chris's fantasy land?
- Oh, no, it is, it is absolutely moving to the edge Chris.
It is going to be a hybrid world where your ML or AI
is going to be deployed, you know, developed in cloud,
deployed on the edge, monitored on the edge,
and maintained on the cloud.
Or it could be a complete cloud native development
and deployment.
So there are so many scenarios, but to your point,
there are a lot of factors that are actually driving
the ML inference to the edge.
Because one, think about few conditions, right?
Number one is the amount of data that is getting generated
is huge, and 5G is actually enabling a lot of data
that needs to get generated.
Because every sensor, every cell phone, every device
is generating a lot of data.
And it's not easy to get all the data to the cloud
and then do your inference there and send it back
because that's a huge cost.
Two, there are applications that are latency sensitive.
You wanna make sure you are able to make the decisions
very close to where the data is getting generated.
There is a need of IT, OT convergence
where you are actually making certain decisions
on how a robot moves or how, you know, the sorting
and picking happens, which is your OT,
now you need to combine that with IT.
And the last but the not least is the importance
with regards to the privacy and security of data
that is getting generated right?
In that context, you wanna do the inference on the edge,
not just inference, you wanna implement concepts
like federated learning on the edge,
where you're retraining on the data that is generated
on the edge and only send the necessary data
back to the cloud, so you can retain the larger models.
But to your point, a lot of workflows are moving
to the edge.
So you will see this hybrid platform going forward,
which is what we call the edge to cloud continuum
that is gonna continue to happen.
- I love that you're drawing in sort of future
thinking around not just the deployment model,
but federated learning and I know tools like
homomorphic encryption can become really important here,
as we think about the next generation
of generating insights from data.
And I'm struck by this one thought that you mentioned
around the difference between machine learning
and model drift, and the more static nature of software.
I thought, well, maybe there's one place
where bit rot is real,
the Y2K bug.
And we had real change in the environment that meant
software actually had to change
to match the new environment.
And, you know, maybe that was the beginning
of DevOps for us.
But I really appreciate your time today Kavitha,
just giving us the sense of, where model drift happens,
why it happens, how we can manage that
and then the importance of inference at the edge.
Thank you.
- Thank you so much Chris.
- You can't expect to train an ML model
and then just walk away from it.
Especially when models are deployed into the physical world
where conditions can change rapidly.
And as we expand the use of predictive models
using real time data, the ability to monitor models
for drift and manage your data and ML pipeline
by evolving DevOps practices to MLOps,
is fundamental to success.
Especially, when you consider distributed deployments
all the way to the edge.