[MUSIC PLAYING]
ERWIN HUIZENGA: Welcome, everyone,
to this session on get faster into production
with end-to-end machine learning operations.
My name is Erwin.
I'm a developer advocate at Google Cloud.
And for this session, I'm joined by Arkid Mitra,
who's a VP of machine learning at Glance.
Arkid will do a longer introduction later on.
So before we go into the Q&A with Arkid,
I want to quickly talk about machine operation,
do a short recap of what is it when we talk about machine
learning operations.
So when we talk about machine operations,
we mean an ML engineering culture and practice
that aims to unify machine learning
system development, dev, and machine learning system
operations, ops.
So let's double-click a bit more on what
does it mean building a machine learning system also for us
before we go into the Q&A with Arkid.
A while back, we launched our machine
learning operations framework.
And when we look at our machine learning
operations framework at a very high level,
there's different phases.
We have the training phase.
And we have the serving phase.
So when we look at training, typically what
happens there is that you do experimentations
and you try a few different models, maybe
some different parameters.
You experiment with that.
And then you hit a training phase or even
the retraining phase if you want to update
your model with fresh data.
And then once you have a model you're happy with,
then you start thinking about serving and bringing
the model into production.
Serving model could web serving or online serving
as an API, model deployment.
And once the model is deployed, you
need to think about continuous monitoring.
Is my model still performing as expected?
Has something changed with the data or with my model?
And then when you have these end-to-end workflows,
of course, you have to think about things
like model management and model governance,
as we heard in some of the other talks as well.
A lot of our customers, users, developers
ask us the question, but if I want to build, manage,
and scale these end-to-end machine learning workflows,
how do I do that on GCP?
This is where a fully managed pipeline service comes in.
This helps you with building, orchestrating, scaling
end-to-end machine learning workflows.
And we do this through offering an easy to use
Python SDK to design your machine learning workflows.
You can use either Kubeflow Pipelines SDK or the TensorFlow
extended SDK to build your pipelines.
This is a scalable service so you can run as many pipelines
as you want and when you need.
It's also very cost effective to use a managed service
like this, because you only pay for your pipelines' run
and the resources used.
And as we talked about, metadata tracking and lineage tracking
is very important for when you build an MLOps system.
So these pipelines will automatically
store metadata about every artifact
produced by the pipeline.
And of course, it's also a secure environment.
So we will be integrating with GCP security services.
[? IAMs, ?] of course, VPC Service Controls.
So managed pipelines can help you build these pipelines
but also manage them at scale.
So enough for the slides.
Let's go into our awesome Q&A with Arkid.
So Arkid, I did a quick introduction of you
earlier in the talk but could you
talk a bit more about yourself, your role, your team, and what
you do with clouds?
ARKID MITRA: So I have joined Glance as the VP of machine
learning last year to build the recommendation
system, and the content generation system,
and the powers of Glance's lock screen.
I previously worked with companies such as Dream11,
XApps, [INAUDIBLE] Data, [INAUDIBLE],,
and a number of machine learning problems with them.
Glance, for those of you who don't know,
provides log stream content to millions of Android phones.
With over 170 million consumers consuming
the content on Glance, it becomes
extremely important for Glance to learn the right content
at the right time at the consumer's device.
The problems to build off at Glance
remains unique in the sense that we do not
have another company like us, and the user growth and scale
is just tremendous.
So that's a little bit about me, and about Glance as a service.
ERWIN HUIZENGA: Well, thank you.
Thank you very much.
You already mentioned a few of the use cases
that Glance works on, and your team works on.
But could you talk a bit more in detail
about some of these use cases and the ones
that your team work on, and how you use machine learning also
in these use cases?
ARKID MITRA: Yes, so one of our primary use cases
is around multimodal personalization
for more than 150 million users.
If that was hard enough, we also have
to find out the causal effects of user behavior,
some amount of synthetic content generation--
and all of this at very, very large scale.
The uniqueness of the Glance problem statement
stems from the fact that content and recommendation
is targeted at a very large population, which has
never used the internet, ever.
And then, there is another section of population which
is using internet every day.
So we have to cater to the needs of all such people.
So that adds that additional bit of complexity.
ERWIN HUIZENGA: Well, that's very interesting.
And it sounds like a very complex problem
to solve for your team.
Could you talk about-- so currently, you're
working on a migration to Vertex AI.
Can you talk a bit about this project
and why your team is using Vertex AI to solve
some of these problems?
ARKID MITRA: Yeah.
For our entry-level systems, we were using an in-house system
for the last four or five years that served us really,
really well.
But as the scale of the company increases,
we realized that we need something more scalable, which
is where we decided to move to Vertex AI.
As of yesterday, we just completed our migration
to Vertex AI.
And all our machine learning pipelines, as well as
the prediction services, have been completely moved
to Vertex AI.
This process was a little long.
It took us a few weeks or a couple of months, I would say.
And we worked very closely with the GCP team
in getting these hundreds of services ported onto Vertex AI.
And there have been very few problems
that we faced around the same.
ERWIN HUIZENGA: Wow, that's a big milestone.
And I would say moving hundreds of models and projects
at that scale in only a few weeks or a few months--
I think that's very, very impressive of you and the team.
Could you talk a bit more about your machine learning
operations approach at Glance?
ARKID MITRA: The MLOps approach at Glance has been a buy
and build in the sense that we have subscribed
to Vertex AI as part of the GCP offerings,
but then built certain components on the top of it,
which makes our lives easier.
GCP team has been working collaboratively
with us for the last few months.
And the basic idea will be to get everything at one place
so that everything related to ML scientists' productivity
is completely self-serve, and people
don't have to move around to multiple services
to do anything.
ERWIN HUIZENGA: Awesome.
Also, you recently have written a great blog post
about recommendation system.
And you talked about why they are so different to operate
and operate into production.
Can you talk about some of the things
that you mentioned in your blog post?
ARKID MITRA: Yeah.
So recommendation systems in general
are difficult machine learning problem statement, right?
And add to that the user scale and the content scale--
it's a perfect recipe for disaster.
If MLOps is hard, what I call [INAUDIBLE] ops is just,
piles up on the top of that.
So you have slow-moving metrics.
The relationship between the offline metric
and the online metric do not come up.
The number of systems--
email systems and banking systems
which have to work together is a lot.
And of course, at the end of the day,
there is an immense amount of compute that goes in.
And add to all of this, if there's
a small problem in your recommendation system,
you are going to kill off your business slowly
without you even knowing that your business is going down.
And it is just cutting off a couple of percentage points
from your revenue every day.
So that adds that additional complexity
of having some monitoring system, some alerting system,
which can alert you, oh, something is going wrong.
So yeah.
ERWIN HUIZENGA: [INAUDIBLE] ops--
I love it, Arkid.
That's definitely one we're going to re-use more often.
We only have a bit of time left.
Is there one lesson learned you want
to share with the audience?
ARKID MITRA: I think one of the primary lessons
would be, be ready for software engineering practices
to hit the ML developers.
If they haven't already hit you, they are going to hit you.
You can expect that the best of MLOps practices, the Git best
practices, precommit hooks, data versioning, pipeline
versioning-- and a lot of all this will come to you.
Because till now, ML was in its infancy.
Now, we are hitting production.
So while not all the software engineering practices
will make sense, some of them will.
And then, we will have to build some additional practices
of our own.
While there will be some inertia right at the start for the ML
teams to move to these practices,
but after a couple of strings, most teams will see the point.
ERWIN HUIZENGA: Love it.
Well, thank you, Arkid, again, for joining us.
It's been a pleasure hearing you talk about use cases.
And big thanks for all of our viewers
for viewing this session, and hope to see you next time.
ARKID MITRA: Thank you, Erwin.
[MUSIC PLAYING]