SPEAKER: So far, we've talked about how Kubeflow Pipelines
bring orchestration for complex ML workflows,
giving you reproducible and reliable experimentation.
And in our last episode, I showed you
a prediction model ready to go.
Let's walk through building a Kubeflow
Pipeline that uses this model.
[MUSIC PLAYING]
Today, we'll go through a sample ML app using Pipelines.
Check out the link below if you want
to read more about building Pipelines in a Jupyter
notebook.
I'll use the same model from the last episode, where you'll
forecast how many Chicago crimes are expected
to be reported the next day based on how many were reported
over the previous days.
If you want to see more details on the model itself,
check out the last video.
To create an ML app on Kubeflow, we'll
walk through downloading data from BigQuery,
creating a Kubeflow Pipeline, including Google Cloud AI
platform components to train and deploy the model,
submitting a job for execution, and sharing
the Pipeline with others.
AI platform is Google Cloud solution
to train ML models at scale, host your trained models,
and serve models to make new predictions.
We're going to run this all from a Jupyter notebook
server running on Kubeflow.
On a running notebook, clone the GitHub repo linked below
and access the AI platform notebook.
If you run into any authentication issues
while running the notebook, make sure you're
on the latest version of Kubeflow
and check out the documentation around the user GCPSA secret.
Getting started with the notebook code
will import dependencies and define constants,
including the Kubeflow SDK.
As you begin to create your Pipeline,
first define the required parameters
like the Google Cloud project ID and storage bucket.
Optional parameters include your storage path
containing your untrained model, the output storage bucket,
Pipeline name, and model name.
Similar to the Chicago crimes notebook we covered last time,
start by defining a download function that
will download trainer data from Google Cloud BigQuery.
This time, you'll be using the BigQuery Kubeflow Pipeline
component.
Remember that components include inputs and outputs, which
can become inputs to other components.
This component submits a query to the BigQuery surface
and dumps output to our cloud storage bucket.
Next, preprocess the data and submit a training job
to AI platform.
Remember each step in a Pipeline is a containerized task.
This means that every component must
be packaged as a container.
To do this, you can load a Pipeline component
via a local file or a component URL, like a GitHub commit,
making sharing versions, and reuse easy.
In this example, we call a YAML file
that defines our next component, which submits a training
job to AI platform.
While this is one way to package a component that
makes sharing easier, there are other ways
to do this, like using the Pipelines SDK, which
includes classes and methods for interacting
with Pipeline components.
Inputs for this component include the cloud storage
location of our model, the input parameters
to create a training job, and the container image
URI to run this component.
It outputs a job ID and storage path to the trained model.
The next component will deploy the model with the ID
form our last output.
This YAML file describes a Pipeline component
to deploy our trained model from cloud storage
to an AI platform service.
Inputs include the cloud storage path of the trained model,
and it outputs the storage path containing our deployed model.
Now you can string together all of these components
to define the Kubeflow Pipeline-- just
define the data set storage path and trainer output storage path
and call the download data function, training function,
and model deployment function.
Finally, submit the Pipeline for execution.
The Kubeflow Pipeline system requires an experiment
to group Pipeline runs.
If you don't specify an experiment,
it will use the default one.
It outputs a link to both the experiment
and the run on Kubeflow.
Once you finish waiting for the Pipeline to complete,
you can see it in the Kubeflow console,
plus related logs from the run.
Here we've added a line of code to compile our Pipeline,
so we can share it with others.
Once you run it, you can locate the zip file in your notebook
server and download it.
The archive file can be shared with others,
which lets them run it on their own Kubeflow clusters.
You can upload a Pipeline file to the Kubeflow Pipelines page
with versioning control built in.
Once uploaded, you can create runs from the same UI.
And Kubeflow has a form to fill out input parameters,
so you can start runs right away, giving you reusability
at your fingertips.
Once you've deployed the Pipeline and tested it,
you can share the archive files, so your teams can deploy it
on the production cluster.
In addition to sharing the whole Pipeline,
you can also share the component files,
making steps shareable and reusable.
Stay tuned to learn where else you can run Pipelines.
[MUSIC PLAYING]