Welcome to this introductory video for our
Databricks notebook environment.
If you're already familiar with Databricks and
the notebooks, feel free to skip this
video. For those of you who are new to
Databricks or just want a refresher
we'll walk through some of the important
features of the notebook so that you can
make the most from this course.
The Databricks notebooks you can think
of is very similar to the Jupyter
notebook style
form of computing, however the Databricks
notebooks offer improved security, better
collaboration tools, and some more
functions and features that you'll see
are important in collaborative
environments but also dealing with large
data.
On the left hand side of the notebook we
can see we have our menu bar where we
can go through our file system in the
workspace, connect to git repos using the
repos folders, and also configure our
compute clusters among other things.
In the left hand side of the notebook
itself, we have the title of our notebook
and then we have a number of options in
our menu bars here to perform certain
functions.
I will make note that during the course
if you wish to save and retrieve the
work that you're doing in the labs you
can export your Databricks notebooks
using the export commands here.
If you're moving your code from one
Databricks environment to another, the
DBC archive version of exporting your
Databricks notebooks will mean that you
preserve the file structure and makes it
a much easier way to
pick up and drop off your notebooks. If
you just want the source file or the
python notebook, you can also export
those formats as well.
On the right hand side you can see we
have a number of buttons we can schedule
runs for our notebooks if we wish, we can
share our notebooks with others and the
other two here are the ones that we'll
be focusing on the button on the far
left
is the run all command and that, as the
name implies, will run the every cell in
the notebook sequentially. And then the
cell next to it is our cluster
state. From here, we can launch a new
cluster, we can start and we can
terminate our clusters as we need to.
Your cluster should already be up and
running when you log into Vocarium which
is our lab environment. If it's not
already attached, you will need to click
on the button here and then you'll be
able to connect with a
pre-prepared cluster for you.
Let's walk through this Databricks
notebook and we'll cover creating new
cells, executing code, creating markdown
cells we'll read some data from the
Databricks file system, we'll visualize
some data ,and we'll also install some
libraries as you'll need to do this
during the course.
We also will make use of some of the
Databricks notebook utilities. For
example, we'll make use of some magic
commands and we'll look into what those
are in just a moment.
At the start of all of the notebooks
both the demo notebooks and the lab
notebooks you'll see that we need to run
something called a classroom setup.
The goal of the classroom setup is to
make sure that we've configured our data
sets, our models, and our user environment
so that it's smooth to operate the code
that we have
for our labs and demos.
We're going to use our first magic
command to run this. This is using the
special percent run magic command and
what that allows us to do is to actually
run another notebook within this
notebook. So in our file structure we'll
have an includes folder
and a classroom setup notebook and we're
going to be running that notebook here.
There are a number of ways that we can
run a cell. We can go to the run menu and
run the current cell, you can also see
the hints for the
keyboard shortcuts either Ctrl enter
or Cmd and enter.
We can also see on the right hand side
we can run this cell using the play
button run cell option here. I'm going to
press Ctrl and Enter on my keyboard to
run this cell and you can see the output
just below running the command is
currently what's happening and then
we'll see some of the output of the
notebook as it's being run from this
notebook.
You'll notice that we'll get this you
may need to restart the kernel, don't
worry about this this is just part of
the intended output that we're seeing
you won't actually need to restart the
kernel.
If you've set up your environment
correctly and we'll have set this up for
you beforehand you won't need to
install any of the data sets and so
you'll see this skipping existing data
sets we'll show you in another video iff
you're using these files on a different
workspace how to make sure that you're
ready to go. And then you can see finally
that we have our predefined variables as
well we have a working directory, a
database, and we have a link to the data
sets that we'll be using in just a
moment.
You can see this completed correctly and
it tells us how long this command took,
who ran it, and when they ran it.
Now let's look at creating new cells.
Cells are the way that we inject code,
inject comments, and everything else that
we want in our notebooks and so it's a
very important piece to
be comfortable with.
So we can delete cells by pressing the "X"
button here and we can add cells in a
number of ways. If you see, as we hover
over the different cells we have a plus
button, we can create one above this
particular cell, we can create one below
using the the UI to do this and we can
also do
keyboard shortcuts to complete the same
task. So if we click on the text
we can see we have a flashing
cursor
and that would be if we wanted to you
know write more text in the cell we
would just write it normally just like
that.
But if we want to create cells above and
below using the keyboard we would need
to click out of the cell here and but on
to the white part just below it and then
if we press A on our keyboard we create
a cell above and if we click back and so
that we make sure that this is
highlighted if we press B on our
keyboard we can see we create a cell
below.
You can see that the default code
for this cell that it's expecting is
python, you can also change this to
different types of languages for a
particular cell to be written in.
The Notebook
has python as the default language and
you can see here in the top left that is
how this is configured so we could
change this to one of the other four
languages that's supported in Databricks
notebooks.
For now though, we'll delete these and
carry on. We've already seen how to
execute some of the cells by running our
setup script, but we can also do the same
to run our code. Here we have some simple
Python and so if we press Ctrl Enter, we
can see we get the output displayed here.
We can also use
non-default languages in our notebooks
so if we had a SQL notebook we could run
some Python and we could also run some
SQL notebook in our python notebook just
like this.
We won't spend too much time on this as
our course is focused on just python for
all of our code and so you won't need to
worry about this it's just important to
be aware that there are many features in
the Databricks notebooks that we won't
necessarily be touching on in this
course.
You'll see also in most of our notebooks
we'll have a number of markdown cells to
describe some of the features that we're
looking at to explain some of the
concepts. You can write your own markdown
cells in Databricks notebooks by using
the %md
magic command at the top. You can then
write regular markdown
and then to render this we just execute
this cell as we did other cells by
pressing shift enter cntrl enter or
cmd return
and you can see all of these are
rendered using the markdown library.
Now we're also going to be reading in a
lot of data into this course, and we have
our specially created datasets for you
already pre-compiled and downloaded into
the workspace.
Let's take a look at
importing a dataset from a CSV file
called labeled news catcher dataset.
We have our CSV location which is at the
paths.datasets in our
pre-made
structure and then we're going to grab
and then we're going to grab the CSV
file here we're going to read that in as
a CSV using pandas and then we're going
to display the data set. So we can run
this cell.
And you can see we have the pandas Data
Frame output just below.
We can also visualize data. We're going
to be visualizing a lot of data and
that's always an important part in any
sort of data science or machine learning
application is visualizing the data.
We'll import matplotlib and plot
the topic and the title.
You can see we have our bar chart here,
we have the number of times each topic
has come up.
And if we want to
display some text or some DataFrames
that perhaps are in a complicated way to
view or or perhaps just needs to be
prettified just slightly, we can use an
inbuilt command from Databricks called
display.
And this will pretty print as it says a
larger variety of different data types
including Spark DataFrames and pandas
DataFrames. So let's look at what the
news catch accounts by topic looks like.
You can see we get a nicely formatted
table and we can also do some simple
visualizations as well that are inbuilt
into this display command. This is
helpful if you want to do some quick and
dirty visualizations of our data.
And also you might have noticed that we
didn't actually need to import
matplotlib or pandas or rather we didn't
need to pip install.
The Databricks runtime which is the
runtime that we're currently using for
this notebook, includes within it a
number of python libraries.
Each runtime, as it's released,
contains different types of python
libraries.
Sometimes though we'll need to install
additional ones and for this course
there'll be a number of python libraries
that we'll need to install.
We do the python stall python installs
using Pip with the magic pip command
here.
Now, typically we see many people
downloading and installing the latest
version so for example if we want to
install NLP test we can pip install just
like this which is fairly common.
However, we do want to stress that it is
best practices to pip install a specific
version of a library particularly if
you're going to share this code with
other people so that you can replicate
your results.
So in all of the instances in this
course we associate a version with all
of the libraries that we still install
with Pip. So I'm just going to rerun this
command now you can see that it
downloaded 1.4.0 but we were going to
keep 1.1.0 so let's rerun this command
and it should uninstall
uh everything that we had and then
reinstall
you can see it uninstalled the 1.4
version and then installed the 1.1
version.
Now that that's imported installed, we
can import NLP test
and proceed to use this library just
with all the other libraries that we
have
for our code bases.
Now if you're interested in learning
more about what you can do with the
Databricks platform and with notebooks
check out all of these other fantastic
guides there's plenty more that we can
do with Databricks notebooks however
this is enough for you to get started
with our demos and with our labs in this
course so you can make the most of your
time.
Thanks for watching!