STEPHANIE WONG: The flexibility of Kubeflow
lets you run your ML jobs anywhere you can run Kubernetes
and gives you an entire platform to work with.
Sometimes you don't need the entire Kubeflow platform,
but you still want to use pipelines to orchestrate
machine learning workflows.
Let's look at Cloud AI Platform Pipelines, a Google Cloud
service that lets you focus even more on your ML
and less on your infrastructure.
[MUSIC PLAYING]
If you want to read more about how
to run Pipelines on Google Cloud,
check out the documentation link below.
Previously, I talked about how to work with Pipelines as part
of a Kubeflow deployment, but we can use Cloud AI Platform
Pipelines to get the same user interface
and back-end functionality.
If you don't need the rest of Kubeflow
and are just looking for the ability to work with pipelines,
Cloud AI Platform gives you a quick way
to set up an installation.
On top of that, authentication with other Google Cloud
services is built in.
This lets you focus on building your pipeline
and integrating with other Cloud services
without needing to manage a complex deployment.
Let's start by getting a Pipelines installation set up.
To try it out, head to the Google Cloud console
and use the left navigation to find Pipelines
under AI Platform.
Then click the New Instance button.
Google Cloud will set up a Kubernetes cluster
and then deploy the Pipeline installation on top of it,
but all the details are handled for you.
Once it's set up and running, you'll
have the full Pipelines UI and SDK to start working with.
We'll go back to the Pipelines page
and click Open Pipelines Dashboard to get
to the same user interface.
We can see experiments and runs here, as well as look
at our metadata and artifacts.
Don't forget to check out the previous metadata management
episode, which goes into more detail about it.
So it's still possible to upload a compiled pipeline
through the same user interface, but what if you're using
a notebook for your model?
Here's where we can use another service to help us out--
Cloud AI Platform Notebooks.
In our previous videos, we've talked
about the advantages of running a notebook in the cloud
rather than on your local machine.
Cloud AI Platform Notebooks gives you
a simple way to do just that.
You can create a new notebook quickly and jump
into a JupyterLab interface.
Back in the Google Cloud console,
just click New Instance, and choose an environment.
Since we're working with a TensorFlow 1 example,
we'll choose TensorFlow 1.1.5 without any GPUs
and hit Create.
Once again, all of the infrastructure
is taken care of behind the scenes,
and we're able to just click Open JupyterLab to get
to our interface, where we can create notebooks and execute
them.
Since we're using the same notebook from the last video,
we've already cloned the same repository here.
Code for Pipelines is made to be reusable
so we don't need to change anything
to create an execution.
We will need to specify the same properties-- our Google Cloud
project and the Cloud Storage bucket where
our model will be stored.
There's one more change we need to make.
We have to tell the pipeline's SDK where our pipeline's
endpoint is located.
This is as easy as copying the URL of our hosted pipeline,
which we can get by clicking Settings next to a pipeline
and storing it as a variable in our notebook.
Since the same user created both the notebook and the pipeline's
installation, the authentication is taken care of for us.
We'll then modify the line where we execute the pipeline
and point it to the server and then run the notebook.
Other than that, this is the same pipeline,
so we'll still be using the Cloud AI
platform to do the training and serving of the model.
We're just doing it from a different environment.
Cloud AI platforms can be an easy way
to get a pipelines environment without having to set up
an entire Kubeflow deployment.
In addition, it's easy to integrate with other Google
Cloud services to help extend your ML projects,
like hosting notebooks, model training, and serving.
Try it out for yourself.
[MUSIC PLAYING]