>> You're not going to want to miss this episode of
the AI show where Rajesh tells me
all the secrets that Bing ads
uses to make your ads make even more money.
It has to do with AI. Make sure you tune in.
[MUSIC]
>> Hello and welcome to another exciting episode of
the AI show that
my friend here Rajesh. How are you doing, my friend?
>> Awesome. How are you?
>> Good. Tell us what you do.
>> I'm on Senior Product Manager in Azure ML.
I help my customers get
their data science problems solve through Azure ML.
>> This is a very problem-specific episode.
We're going to go through like a very
specific problem and how to solve it,
the data science where in how AML help.
So let's start with the actual problem.
What are we trying to solve today?
>> So we are working with closely with our Bing ads team,
and we're trying to help them to get a recommender system,
a recommendation report out through
their advertisers to make the advertisers
successful in their digital marketing.
>> So let's see if I understand this right.
It's like we're trying to make
a recommendation system for
people that are putting ads on Bing and we're
recommending to the people putting ads on
how to do their ads so they can make more money basically?
>> Correct.
>> Because when I heard recommender system in ads,
I thought like where are we going to put ads on the Bing page?
Now this is for actual advertisers.
>> Correct.
>> Okay. That's a little different.
>> Yes.
>> So why is it kind of a challenging problem?
>> It's challenging because first of all as a customer,
as advertiser, everybody is
trying to maximize their digital marketing budget.
They kind of come up with a list of
keywords or list of ads that they wanted to actually bid on.
But they really don't know what's going to
work and what's not going to work.
>> Right.
>> They don't end up necessarily
maximizing their digital marketing budget.
>> So let's talk about
the data science problem because I imagine like
if we're recommending stuff to the advertisers,
there's going to be some ginormous corpus.
Let's talk about the scale of
stuff and let's get into the data science problem.
>> Absolutely. Roughly we have about this specific problem
is telling the advertisers which keywords to
use to boost their ads and to boost clicks on their ads.
We basically use live GBM,
we use some clustering and come up with
what other advertisers are doing for similar products,
which ones are getting more clicks,
which ones are getting less,
which one are being seen more often and so on and so forth,
and then we come back with
what is the recommendation for this advertiser.
If they use these keywords,
if they place their ads in this particular way
and use certain texts recommendations,
then they basically get a boost in their clicks and views.
>> Is this what we're looking at on the screen then?
Is this an example of this kind of this report?
>> Exactly. This is what the advertiser actually sees.
So they can actually see that
if they can boost their adds by positioning it,
they can increase their number of
conversions by running it on certain days,
and they can also get
their ads and render more customers by adding new keywords.
So there are different recommendations
we have for the advertisers.
>> That's pretty impressive. So again,
we talked a little bit about the models.
Like, are we talking like gigabytes, terabytes of data?
How much data is actually being run when this is trained?
>> Usually it's in terabytes.
>> Okay.
>> Yeah.
>> So let's talk a little bit about
how Azure Machine Learning Service helps now.
Because obviously there's a lot of
options when you're using Azure Machine Learning Service.
There's a training aspect.
There is a model management aspect.
There's a endpoint actually serving things out.
Which part of Azure Machine Learning Service
do they choose to use and how did it help?
>> So mainly about Bing ads,
customers are using Azure ML for inferencing.
So what they do is they do
their offline training because there's a lots of
data and the data is stored right now in their own datacenters.
But they come back and they say they want to scale up.
They want an easy way to package their model.
They want to scale up, scale down on-demand.
Then they want to deploy it in Kubernetes cluster,
and that's what we helped them do.
So that's pretty much like,
for them, they don't have the right lines of code.
Instead of that, they're just writing couple of
lines of code and getting it done.
Most of the infrastructure, monitoring, scaling up,
scale down, model management,
all of that is taken care by in Azure ML.
>> So let's talk about this a little bit more
precisely because I feel like there's
a lot of concepts that went in there.
Let's talk about primarily what model management is,
and then what it actually means to make a deployment in
Azure Machine Learning Service and
why is it they don't have to think about all this extra steps.
So let's talk about modeling management first.
What does that mean in Azure Machine Learning Service?
>> So modeling management is similar to software management.
When you have a software component,
you are doing versions of your software or
your models and then you're trying to
understand how your model is performing.
If it's not performing well,
what are the reasons why it's not performing.
If you want to basically understand that sometimes it didn't work
well and you've got some kind of
escalations, what really went there?
Was it data issues,
was the training not correct,
or did something happened
within the model that actually went wrong?
So you need to have the versioning,
you need to have the capability of
figuring out what's behind the scenes
and quickly debug that and get something corrected out quickly.
>> I see. Because usually,
I'm being honest here,
I would just make a model and then we put it out on the server.
But now with model management Azure Machine Learning Service,
you're able to say we're using model version 7, for example.
If that model starts performing really bad,
I'll be like, "Hey, everybody.
We got to pull the plug. Let's go to
six while we make eight better."
>> Exactly.
>> Especially when we have like data drift, for example,
which is really easy, or maybe you're
having because this is going to start to happen.
I feel like people are going to start to understand your models
and maybe start to attack the model with the wrong data,
which is like, "So
being able to version model is super important."
So we're talking of models,
now how do we go from models to
this easy deployment that you were talking about into Kubernetes?
>> Sure. So the Kubernetes deployment
is pretty much like a few lines of code.
I mean, basically they'll handed
off to their software engineers to do.
From there on, we basically scale it out.
We take their model, scale it out in
multiple clusters really to
understand how [inaudible] will work in that case,
and then we would make sure that it's ready for inferencing.
For example, our current scale is we do
about roughly 300K inferences per day.
>> Okay.
>> If you had to even provision for that,
you would basically have to first hire,
you're to get hardware in,
you're to provision the hardware,
you are to basically make the container packages
correct on all the versions of the compatible softwares.
We do all of that,
and currently this is being
consumed by close to half million advertisers.
>> I see.
>> So the scale is huge and basically it takes away
the load of doing the infra and
the platform part from the developers and data scientists,
and they focused on the business results more.
So now they can do more of such capabilities for
their customers to make them
successful versus focusing on the infra.
>> Got it. So I imagined that in Azure Machine Learning Service,
there's going to be some button that
says this model I want to deploy it?
>> Yes.
>> When you do that, what do you have to tell it?
I mean, let's just say I have
a raw protocol file that I trained in TensorFlow,
there's usually some code that I have to load it up.
Let's just say I have a PyTorch,
PTH file that I load up but I know how to exempt.
I'm more familiar with PyTorch at this time.
Usually, there's some code that I have to
write to do the inferencing.
Are you like hijacking that from me or
where's that code coming from to do that inferencing?
>> The code is pretty much like pre-builtin.
So basically what their doing is you're giving us
your model and then we are
basically packaging it and creating
a web service out of it and deploying it.
So you don't have to do anything on your side.
You're just saying, "Hey, just deploy this model."
So it's basically a few lines of code with some parameters.
You will give a scale, how much scale do you want
on the inflow and the resources, and you're done.
>> That's pretty amazing. Yeah, I kind of see them up on that.
I was like I've done this before.
Basically, you just have to say
what environment and how to score it,
and that's basically it. It's pretty amazing.
I've deployed stuff out to Kubernetes,
and like I said Kubernetes is
hard and it's really nice to have it managed.
Is there anything you would like to add?
Where can people find out more about this?
>> Sure. I think I would say there are
a lot of resources available on azure.com and azureml.com.
So go to the website,
look at tutorials, lot of code on GitHub.
Try it out yourselves,
give it a shot, and we're here for feedback as well.
If you find something that is missing,
happy to get feedback and do better on that.
>> Fantastic. Well, thanks so much.
Thanks so much for watching. This has been
another episode of the AI show.
We've learned about how to deploy
models out using Azure Machine Learning Service.
Thanks so much for watching
and we'll see you next time. Take care.
[MUSIC]