[MUSIC PLAYING]
PRAKASH KAMATH: Hey, Brian.
BRIAN SUK: Oh, I didn't see you there.
Oh, you scared the heck out of me.
Sorry.
I was just busy--
focused-- trying to build this sculpture
with all these building blocks that we have.
The problem is is it's so unstable,
and it keeps falling over, and I spend so much time
trying to maintain and build these things.
PRAKASH KAMATH: I know someone who
has just the same problem with compute resources.
They tried Hadoop, but it fails often and is super expensive.
BRIAN SUK: Ah, there must be a better way, some sort of tool
that builds all these things for us and maintains it,
so we can spend more time analyzing and doing
the things that we love, instead of spending all of our time
building and maintaining.
PRAKASH KAMATH: Hmm.
TOGETHER: Dataproc.
PRAKASH KAMATH: You know what?
I have just the thing.
BRIAN SUK: Nice.
[MUSIC PLAYING]
PRAKASH KAMATH: Let me show you how
to create a large Dataproc cluster with preemptible nodes
using the Google Cloud console with just a command
line in minutes.
First, we'll create the cluster.
We'll give the cluster a name.
Then we'll pick a place to put it.
We are in New York, so we'll pick US-East.
Next, we get to choose the cluster mode.
And there are three choices.
For this demo, we'll go with Standard.
After that, we get to pick the shape of the master node.
We'll go with 2 vCPUs.
We'll set the primary disk to 10 GB, which
happens to be the minimum.
Then we get to customize the shape of the worker nodes.
And you'll see, it's really easy to customize the shape.
And this was really important for a customer of mine.
We'll set the primary disk size to 10 GB, again.
And we're creating a really large cluster
with preemptible nodes, so let's configure the preemptible nodes
next.
We'll set the number of nodes to 400,
and the shape of these nodes will match the primary worker
nodes.
Now, we could create the cluster right to this user interface,
or what we can use is the REST command that it auto-generates,
or even the gcloud command, which is also auto-generated.
Now, because I want to measure the time it takes
to create this cluster, I have a little script
that we'll run using Cloud Shell.
And that'll create the cluster for us today.
Now, as this cluster is being provisioned,
we can actually track the progress
through this user interface.
So we could go up and click on the Clusters,
and you'll see about 400 nodes being provisioned as we speak.
You can click through and go to the Details.
Look at the VM Instance list, and you'll
see all of these preemptible nodes getting provisioned.
Now, a few things about you using
preemptibles in your cluster--
they help you run very large clusters at a low total cost.
The machine type of these preemptible nodes
matches the machine type of the primary workers.
Now, within a Dataproc cluster, preemptible instances
are added to a managed instance group.
And as you know, they can be reclaimed at any point of time.
So these instances can be re-added back
into the cluster if and when capacity is available.
Also, a few things to consider about preemptibles,
they're used as processing nodes only and do not store data.
By default, preemptible nodes are
created with the smaller of 100 GB
or the size of the primary node's persistent disk.
And this is, again, used only for local caching
and is not available for HDFS.
Also, you cannot create a Dataproc cluster with
preemptible-only nodes.
At minimum, a Dataproc cluster will have two primary nodes.
This ensures that the cluster will never run out of workers.
You can add or remove preemptibles
from a cluster as needed, and even
set the number of preemptibles in a cluster to 0.
50% or lower of the total number of worker being preemptible
is a good working ratio.
As you can see, the cluster is now ready for use,
and it only took a couple of minutes.
And you can even submit your jobs
through this user interface.
[MUSIC PLAYING]
BRIAN SUK: Oh, thank you for showing us.
That's really cool.
So what you're telling me is that
with a few clicks of the mouse, we
can easily set up these large clusters
that build and heal themselves.
Also, I really do love that it's super quick to do,
and it doesn't take ages to set up and build,
like it did when I used to have to do this by hand.
But my blocks, my poor blocks.
Don't you have anything out there
that will build this for me?
PRAKASH KAMATH: Well, Brian, we can't build your blocks
for you.
But what you just saw were the building blocks
for big data analytics.
BRIAN SUK: That is really cool.
Thank you so much for watching.
And remember to click Subscribe and share this video.
PRAKASH KAMATH: And if you're working on any cool projects,
connect with us by clicking on the link in the description
below.
[SINGLE XYLOPHONE TONE REPEATING]
[SLOW SWOOSHING]
[MUSIC PLAYING]