SANDEEP PARIKH: Hi, folks.
Welcome to the next episode in our series
on Modern DevOps practices.
Today we're going to talk about building software delivery
pipelines.
We're going to go through the tools, the processes,
and the practices that you can use
to build software delivery pipelines using Google Cloud
Platform.
My name is Sandeep, and I'm a DevRel engineer
at Google Cloud.
I've been here for almost seven years minus some time away,
and I help teams adopt and optimize for cloud.
You can always find me on Twitter @crcsmnky--
that's "circus monkey"-- on Twitter
if you've got questions about DevOps, Google Cloud,
or whatever that comes up.
So many of you have probably seen the four key metrics
already if you've been following our Modern DevOps series.
These are trailing indicators that we
can use to measure software delivery performance.
There are two speed metrics and two stability metrics.
The speed metrics include deployment frequency
and how long it takes for a commit to go into production,
and then the stability metrics include how many changes we
ship that include failures and the time
it takes to restore service after a failure has
been detected.
Now, when it comes to building software delivery pipelines,
it has a pretty direct and somewhat indirect
impact on all four of these key metrics.
But again, these are just trailing indicators.
So how do we know if we're making
progress along that path?
Well, the DORA research gives us a lot of detail
around how to drive organizational performance
using software delivery and operations.
And one of the key capabilities included
there is continuous delivery.
And as you'd imagine, software delivery pipelines
are a big part of any continuous delivery process.
And those also include and sort of
are driven by a bunch of additional capabilities
on the left side, so things like streamlined change processes,
having additional autonomy, loosely coupled architecture,
and so on.
And ultimately what we find is that these are great leading
indicators of success, and they have a really positive impact
in terms of reducing toil, the pain of deployments,
and most importantly, burnout.
Now, the software delivery process
is typically comprised of three main buckets--
the code, build, and run phases.
Now we know that this is really a coarse definition of software
delivery.
There's quite a bit more that goes into the delivery process.
So let's explore that a little bit.
Now, when we think of Code, Build, and Run,
if we map that to Google Cloud's offerings,
we actually have a lot more that's going on there.
So starting in the upper left, around Develop,
we have Cloud Code.
And then for building, testing, and packaging our artifacts,
we have Cloud Build.
And then, there, you can kind of take two paths.
You can either deploy directly from Cloud Build or deploy
using Cloud Deploy to things like GKE or Cloud Run.
Or after building, you can just store your artifacts
in Artifact Registry, Cloud Storage, or any other artifact
mechanism like Artifactory, npm, Maven, or GitHub.
But ultimately, this is sort of the superset
of that Code, Build, Run approach
as it maps to Google Cloud.
For this talk, I want to start by focusing
on just the building aspect-- building,
testing, and packaging using Cloud Build.
So let's dig in a little bit.
Cloud Build is a managed service and really kind
of a workflow mechanism for building, testing, packaging,
and deploying all manner of software.
Now, Cloud Build has a number of really important features,
but I think the really key takeaways are the fact that it
is a fully managed build service and it uses containers
to execute its build pipelines.
And you can pull in the source for your Cloud Build workflows
from any upstream source, whether it's
Google Cloud Storage, Cloud Source repos, GitHub,
Bitbucket, or GitLab.
And you can set these things to trigger based on branch or tech
commits or when you're just doing things
like GitHub actions.
Now, the nice thing about Cloud Build is that, in many ways,
it's a general-purpose workflow mechanism, which
means you can actually build whatever you want
inside your build pipeline as long as each build step
can execute as a container.
And you can deploy to a number of downstream services,
whether that's deploying artifacts into artifact
registry or to Cloud Deploy to hand off there,
or directly to things like GCE VMs
or Cloud Run, Cloud Functions, et cetera.
Now, I mentioned that Cloud Build
is a general-purpose workflow mechanism in many ways.
And what we find is that the use cases are pretty large.
It's not just about building web applications.
That process is obviously there, but you can actually
use it for a variety of things, so starting
with general continuous integration
and build capabilities, you can use
it to run integration tests, you can actually
use it for deployment automation and infrastructure
as code as well.
So you can run things like Terraform pipelines
using Cloud Build.
And it actually has a number of benefits
for doing data workloads or ML workloads as well.
You could actually imagine building your Dataflow Java
packaging there and then deploying that
to the Dataflow service.
So it has a lot of validity and utility
for a lot of different services.
Now, the Cloud Build workflow is pretty straightforward.
You start with your source code, you
get a Workspace to build as part of the service,
and then you define each of your build
steps, again, as a container.
And that final build step determines
what happens to your built artifact,
whether it gets deployed to a running service
or it gets saved in some kind of a registry mechanism
like Artifact Registry or Artifactory.
And the nice thing about Cloud Build
is that it allows you to really dig in and adopt
one of the key DORA capabilities, which is loosely
coupled architectures.
Because Cloud Build is meant to be
a very loosely coupled mechanism and a workflow approach.
So you can run individual build steps
as kind of isolated processes and then only
hand off the components you want to to the next step.
So you can imagine a very complex workflow
that includes multiple Cloud Build pipelines
or just has different sorts of branching and approaches
for actually building your artifacts
and getting them ready for deployment or for production.
And those pipelines, again, very robust approach here.
So you can do things like kind of paralyze multiple test
processes.
Like maybe you have certain integration tests
that run in one part of the pipeline, and at the same time,
maybe running other unit tests.
And then they can converge with both
of those as a waiting condition into that final step
for deploying.
And when it comes to triggering builds,
you have a number of options to kick off
that Cloud Build workflow process.
You can have them driven by repository triggers,
a manual approach, or even a scheduled approach,
or we could even get really fancy
and use things like Pub/Sub queues
to drive your Cloud Build pipelines to start.
So you have a lot of flexibility and capability
when it comes to how do we actually get our build started
and then what do we do when it's done as well.
Now, here's an example of a very simple Cloud Build file.
And this defines a series of steps that actually take us
through building a Docker container, pushing
that container to a Docker registry,
and then actually deploying that container to a GCE VM
so we can create the VM and actually deploy using
that container that we just built in the previous steps.
And so this, by itself, would actually take a simple package,
build it, and then deploy a VM with that container running
on it right out the gate.
And when we talk about those build steps, those container
images that are your build steps,
Google provides a number of handy build container steps,
so things like docker commands, building go applications,
using the gcloud CLI, gradle or maven for Java applications,
kubectl for orchestrating Kubernetes clusters
or orchestrating Kubernetes applications,
and even things like npm.
And that's just what Google provides.
If you look at what the community has provided
in our hosted repo, there are a number of additional containers
that you can use as your build steps.
Or you could actually create your own
and build them yourself.
Now, here's another example of a Cloud Build pipeline
that actually doesn't have a defined pipeline approach.
So if you've just got application code and a Docker
file, Google Cloud Build knows exactly what
to do with that it.
It knows that you just want to simply build a Docker image out
of this, and it will follow some basic assumptions
based on what you give it.
And if you give it a tag, it'll actually put it
in Container Registry for you.
So even without a cloudbuild.yaml file,
we can actually take a basic Docker image approach
and build that container for you.
And as I mentioned earlier, among the many use
cases that Cloud Build supports, one of them
is also this idea of using Cloud Build's workflow approach
as a tool for doing infrastructure automation.
So we actually have examples on the Google Cloud Docs
around using things like GitOps and Cloud Build
to automate infrastructure deployments that use Terraform.
And so you could actually have deployments
that have a repo that's upstream that's got
a cloudbuild.yaml file that specifies your Terraform
workflow and not only does the plan but also the apply process
as well.
So it will actually automate a lot
of the way you do infrastructure.
And I like this approach quite a bit
because it allows us to get out of the realm of manually
or imperatively doing Terraform operations
and instead making it come out of the service
like Cloud Build can give us.
Now, here's an example of a cloudbuild.yaml file
that just simply does a Terraform plan operation.
So you can see it in the Name field.
That's the Terraform Docker image
we're going to use to execute this build step.
And then, as my arguments, I have a little script there
that's actually going to tell me what
to do based on the environment I want to deploy in.
And ultimately, this is going to run Terraform plan based
on the repo that it came from.
Now, building is obviously just one part of the software supply
chain, but it's a really critical part.
A lot of the work really converges
around getting that source in, buildings those artifacts
to get prepared for deploying and ultimately running
on a compute resource.
So when we talk about the other steps in that flow,
our next thing we want to cover in this talk
is around storing using Artifact Registry.
And this is where we store artifacts, as you can imagine.
Now, Artifact Registry is a scalable, reliable, and secure
space to store your artifacts.
And those artifacts can be many things.
They don't have to just be container images.
They could also be Java packages, Go packages,
Python packages, or even Node modules.
And the list kind of continues to grow, on and on.
So think of this as the place to store any artifact
that you want to use later on.
Now, the artifact management capabilities
kind of come into three main buckets.
There is the storage, analysis, and then the sort
of delivery aspect as well.
And it's all done in a way that maps really well
to the rest of Google Cloud's infrastructure.
So Artifact Registry is tied closely into Cloud Build
and it works really, really well when
you're trying to pull those things like Docker images
into Kubernetes clusters or into Cloud Run services as well.
And it's a great way to kind of build a dependency management
approach that we'll talk about in a little bit as well.
So when we talk about the artifact delivery aspect,
artifacts that are stored in Artifact Registry
can be pulled in from any downstream service--
Cloud Functions, Cloud Run, GKE and GKE Autopilot, as well as
Compute Engine.
And as I mentioned earlier, it really
is kind of a universal artifact management approach.
We think of Artifact Registry as a single place
to organize not only, again, just container images, but also
your language packages and your software dependencies as well.
So as you're thinking about bringing your dependencies
in to GCP and hosting them there,
Artifact Registry is a great place to do that.
Now, when we talk about the storage aspects,
there are a lot of fine-grained controls
you can exert on top of the storage mechanism.
So there's a lot you can do with identity and access management
about who can store and who can pull from artifact registry,
down to even individual repos.
So you could say, for our Docker containers,
we only want to allow our Cloud Run service or our GKE services
to pull those Docker images from our Docker storage repo
inside of Artifact Registry.
Or you could actually do the same thing
in terms of your application dependencies.
No bare application dependency could be pulled into a compute.
It can only be pulled in by another Cloud
Build or another Artifact Registry component.
And with the storage aspects, we also
try to make sure that your artifacts are
really close to where your applications are
going to be running.
So we have regional repositories you can deploy all
across the globe in any of our regions
so that your artifacts are really close to where they
need to be for quick delivery.
Now, analysis is something that we talk about a lot
but we don't actually get to spend a ton of time on it.
And we won't go into too much depth today,
but ultimately what we find with the software supply chain is
that dependencies can be their own potential vector for attack
or things like container images as well.
So what we do as part of Artifact Registry
is that we have automated analysis capabilities.
So we're constantly checking all of those packages,
your Docker images, and all of your language artifacts
to make sure that there are no known vulnerabilities contained
within them.
And if there are, we will highlight that and show that
to you as quickly as possible.
And you can actually plug this into other existing tools
to expand on those detected vulnerabilities
and take different actions that you may need to,
such as blocking deployment for any software packages
or container images that have known vulnerabilities in them.
Now, I mentioned this a little bit earlier,
but I want to expand on it just a little bit.
When we talk about artifact registry--
and this fits in with Cloud Build and the loosely coupled
approach--
you could have an ability where we could say,
instead of building our huge, monolithic application in one
build workflow, we could actually
do a loosely coupled approach and instead
build individual capabilities as their own language packages
and then use those as dependencies for other Cloud
Build workflows so that what happens
is you have shared code, it gets built by Cloud Build,
and then deployed and stored as an artifact.
So that dependency is stored.
And then you may have application code
that pulls in that artifact as your Cloud Build pipeline
and then proceeds through the process.
And this loosely coupled approach
lets teams iterate at their own pace
and maintain their own sort of release velocity
without slowing things down, and it
allows us to narrow our view when
it comes to artifact analysis so that if we find
a vulnerability, we know it's not
across the entire application, it's
just in this one dependency that was
built in a separate process.
Now, here's an example of a Cloud Build file
that actually stores an artifact into Artifact Registry.
Now, I've skipped over some of the setup steps here.
And ultimately what I had to do was set up
Artifact Registry with a Maven repo
and then include the Maven repo components in my Maven
build file.
But once I've done that, I can actually
create a simple cloudbuild.yaml that
can do steps like running my tests,
packaging of my artifact, and then deploying
the artifacts to the Artifact Registry as well.
So we have all of that kind of built into the platform
and it makes this process super easy.
We've actually spread out some of the complexity
here by putting some of the details in the Maven POM file
and some of the complexity in this Cloud Build YAML file,
but it makes things really clean and easy
to evaluate and understand.
Now, that's all I wanted to cover today.
I know this was a quick tour through building software
delivery pipelines, but we wanted
to give you a sense of what's possible with Cloud
Build and Artifact Registry.
See y'all in the next episode.