I'm not going to say a specific place,
we just have to look for the "Data
Exploration and ML Modeling" notebook
using the New York taxi data set and Spark MLlib.
Once you find this one,
you can click on it and click "Continue".
What you see here is that that exact notebook that
we're actually going to run and walk
through on the Spark pool that we just created,
that is a notebook that you'll see here.
You'll get a chance here to actually explore these,
which is an awesome feature.
You'll be asked to explore and say, hey,
maybe this wasn't for me or, yeah,
this is the sample that I was looking for.
Once you're happy and found the right sample,
which we have in this case,
we're going to go and click "Open notebook".
Now what's going to happen once I click this
is that it's magically going to take
this notebook and open it for me
in my "Develop" hub in my workspace.
Let's do that and let's watch that magic.
You see here? I ended up in my "Develop" hub;
that's the one that's highlighted here.
The notebook has been opened for me; it's ready.
The first thing that we want to do in order to be able to
start running it is to actually
attach it to the Spark pool that we just created.
Remember, mine was called "MySpark3Pool",
so I'm going to select that one.
Actually, I'm going to kick off and start
the session while I'm talking here.
Because the first cell that you run, the first time you
it's going to start the Spark application and start
your Spark session for you.
What that means is that it's going to require a couple of
minutes to warm up and start this session.
It's a good thing to know in case you wonder, hey,
why do I have to wait a little bit before I
can start running the other cells. This is the reason.
This is how Spark anywhere operates.
Initially, you need to start up the session.
There are a couple of minutes of costs
that you have to pay there in terms of waiting.
But while we're waiting for the session to start up,
let's take a look at what we have in the first cell,
and the notebook, and what we're trying to do.
In this notebook, we're going to
leverage a New York Taxi Datasets.
We're going to do some Data Exploration
and some Machine Learning Modeling;
we're going to train a Machine Learning model
after we've done some featurizations.
But the first thing you see here is that we're actually
importing a set of libraries.
This is very common as you're working with, really,
any type of workload,
I would say, when you're using Spark,
is really that whether you're doing Data Engineering,
or just some Data Exploration,
or even Machine Learning and Data Science,
you're likely going to need
libraries that you're going to import.
In some cases, in this case, for example,
we're actually going to leverage a lot of
the built-in libraries that come with the runtime.
A good positive aspect
of a runtime is that we've actually gone and
analyzed which are the most common, frequently used libraries
that a lot of users have and want to use every day;
we made sure to include those in our runtime.
It means that for most task,
especially the ones where you find notebooks and
other samples in the
"Knowledge Center" that we just went to,
you probably don't need to add
additional libraries to your session.
But as you work with Spark and
let's say you use PySpark, for example, a lot,
or Scala and you want to
do different exploration that requires library's
One way to do that,
I'm going to show you this way.
You saw that there's a little setting
that pops up a pane here that appears for your session.
If I would want to,
let's say, install my own libraries,
let's say there's a specific library that
I noticed when I import it doesn't exist,
it gives me an error so I need
to add additional libraries,
which is super common when you're
working in these Notebooks,
is that you would need to go to
Configure Session and on Packages,
and the reason that I can select
Packages here is because we
enabled the setting earlier
when we were creating the Spark pool.
We said we want to enable
the session level package management,
which essentially allows you to then upload
a file here, a.YAML file.
I'm not going to go into the details of that.
You don't have to know that right now,
but just good for you to know,
like the takeaway from this is
know that you can install your own libraries,
other libraries than the one that are included,
and this is how you do it for this session,
which is really convenient in my mind.
You just do it right from
your Notebook and then once you restart the session,
you'll get those loaded for you.
It's good to know that if you click here,
you'll get the properties of the session on the Notebook.
It looks like the first cell ran successfully.
You see that we were importing
libraries like notplotlib in this case,
we're going to do some plotting.
That's a really common library to use,
and we also import some libraries from Spark MLlib,
and we're going to actually use
Spark MLlib to train a model,
and the good thing about that library is that
it's a very Spark specific library,
and it actually distributes the training and
leverages the full power of Spark,
the distributed training on Spark, which is great.
But you can also use
other libraries if you're running here, you can use,
for example scikit-learn or other types of
libraries if you're doing
some modeling and data exploration.
Now, let's move on to the next step, Ingest data.
Because we just did the first one.
We started up the session,
we looked at the libraries and now let's get started
for real and one really cool feature actually,
if you want to know, hey,
I don't even know what sections my Notebook has,
like how many steps, how many sections do we have here.
This is one of my favorite things,
where you can quickly actually get
an overview of the table of contents in your Notebook.
You can see here that the first thing
we're going to do is to Ingest,
and then we're going to do some Exploration,
some Featurization,
Encoding, and then in
the end we're going to Train a Model,
and Evaluate the Model,
just a little extra tip.
Ingestion of data. The data
set that we're going to use for
this particular sample is the New York
taxi yellow cab data set.
The good thing about this is that it's an open dataset,
and it means that I don't really have to
go and manually download a file from somewhere,
upload it into the Lake and work with it this way,
I actually can do it right from this cell.
Now this dataset actually contains,
I think in total around two million records,
and if we use the entire dataset,
we're going to sit around and wait for
some time every time we run one of these cells.
You should actually run this cell
by the way, I forgot to say that.
One of the things we're doing is that we're downsampling.
We're going to downsample this to seven days of
data and work with that.
But it actually means that, I think,
it's still the seven days of data for
this particular sample is still a lot of data.
So what we're going to do is we're actually going to
downsample it even more,
and that's what's happening in the next cell.
You see the first cell here,
it downsamples to the seven days that you
see between these date intervals,
and then we downsample it even more,
and after running this step,
where we downsample it to a fraction of the previous,
we're going to get around 2,000 rows,
and you see here that there's really no way for
us here to actually see how many rows were working with.
I'm going to ask you to follow along with me here.
Let's add a code cell.
Let's be a little brave and add some code ourselves,
and actually go and check
how many rows this dataset has.
For that, I'm actually going to
copy the name of the DataFrame,
and I'm going to put it down here.
If you create a code cell and you just add this,
and then you put dot and you type "Count".
If you do this and you run this with me,
it's optional, you don't have to,
you can just follow along if you want to,
this bonus cell that
we added here in our sample Notebook,
I just want to see how many rows we're
working with, because it's helpful.
I wasn't lying, it was around
2,000. It makes me feel great.
Now if I want to actually explore the data,
which is also very common to say, "Hey,
I actually want to take a look at this data
because I actually want
to see what columns I'm working with.
Like, what is this data set?
What type of columns does it have?"
For that, I can do the same thing,
I can type "Display" and within the display call,
I'll just put the DataFrame,
and this is something to
remember if you haven't worked a lot with Spark,
is that it's actually super easy to say, "Hey,
I just want the top 10 rows
in this DataFrame, just show me.
I'm not necessarily looking at everything.
I just want to take a look
at what does this even contain?"
What you see here then is that it's
going to display top 10 rows for us,
and you can see here that this is a taxi dataset.
It has the pickup time, day time,
it has a drop-off day time, number of passengers,
the trip distance, and a lot of
other settings that you see here on fare amount,
payment type, and all of these things.