Big data is big. Just how big you ask?
Well Adi Polak explains how Spark ML on Azure can be used for machine learning
at scale, using big data.
Adi also covers how spark can be used in AI for good,
a public charity that solves large scale social issues with artificial
intelligence.
And finally explains how a full end to end machine learning solution can work
with Azure.
Let's start with just one product.
What's your favorite product in the machine learning category or family of of of
things on Azure?
Absolutely.
So I really like a Spark ML and you can use it on different products that we
have. So if you work in with spark,
you can work with HD insight or you can work with data bricks and all this kind
of stuff.
And right in there you can integrate spark ML to do machine learning at scale
and they even have this library named tensor frames,
which is actually tensor flow with a spark API.
So it's built on top of spark data frames,
really fascinating and really helps to, you know,
get things done faster because build on top of something that already exists.
Um, yeah.
So I'll just roll back a little bit. What's spark?
That's a good question. So spark is analytics, uh,
framework to do, um, they data analytics at scale.
So it's a distributed frameworks to do analytics at scale.
You can do batch processing, you can do stream processing, uh,
depends on what you need.
And you can also connect it to different products on Azure,
like, like ML,
like ML, like cognitive services, um, and all those kinds of things.
Okay. So spark ML is,
is that one product or is it like a family of things like can you do more than
just one thing with it?
So spark ML is a library that's Apache, it's open source by Apache spark.
Um, and you can do a lot of different things with that.
They don't implement all the machine learning algorithms because there is still
a challenge of distributing some of the machine learning algorithm just because
the way they are built. Um, but they keep on adding new stuff into it.
Um, and it's pretty interesting. And what they're doing now,
they're actually building kind of, um, engine,
kind of a framework that other companies can create their own open source or
other communities can create their own open source. Uh,
so they will be able to run it on top of spark and you can run all of that in
the cloud.
Okay.
So what are some of the use cases that people or companies or whoever is using,
let's say spark ML for [inaudible]? Sure.
Um, so all machine learning that you can think of, let's say, um,
AI for good. So recently, yeah. So recently, um, they were looking into,
um, there was the situation where white leopards, um,
they thought they were extinct and they wanted to make sure they're, you know,
they have the right numbers of how many white leopards exists.
So the thing you have to scan like millions of millions of pictures and for that
you need a lot of, a lot of people to go over those pictures. Um,
but you know, you don't want to do that. You want to use machine learning,
you want to do image processing to figure out how many exists there.
Cause even in some of the pictures for us to figure out there's a white leopard
there, it's really hard on the background. Um,
like the Sandy backgrounds. Cause it's kind of like you don't see them.
Now that's kind of the whole idea of a leopard isn't it? You don't see them.
So yeah. So how does that work? Does, so that's machine learning.
You train a model and that's, it does that or so you train a model,
you class it basically build a classifier. Um,
they connected with bing so you can go online and search for photos.
So you can actually collect photos from bing knowing that it's a white leopard.
Um, yeah.
And this is how they build their data sets and later on classification and they
running at and um, it really helps doing, uh,
doing things a bit different rather than having a human being sitting there and
classifying photo by photo. And it might be a little bit quicker,
I would imagine. A little bit. So how do I,
how would I get started with, say, spark ML or using AI in general?
So depends, you want to use the cloud. Um,
so for spark ML you would need the infrastructure.
So there's a few products that support that. Uh, you can go with HD insights,
uh,
and then deploy your spark cluster and then basically import spark ML library
and start using it from your code. You can also go to a different product name,
Azure data bricks and an Azure data bricks.
It's basically a sass product to run apache spark workloads. Uh,
so there you also do an import saying I want to use spark ML or you can
configure their specific runtime for doing machine learning, uh,
which is quite fascinating cause they already, um,
make made sure everything is performing well for machine learning purposes.
Um,
so you can pick that and it's already has building Python and all the different
libraries, so you won't even need to import them. It's already there.
[inaudible].