MATTHEW SOLDO: Hey, everybody.
Welcome to my talk about buildpacks on Google Cloud.
My name is Matt Soldo.
I am a product manager at Google focusing on our serverless
compute platforms, Cloud Run, App Engine, and Cloud
Functions.
And for the past few months, I've
been working on bringing Google Cloud's buildpacks to market.
And I'm really excited about talking to you about them
today.
So what we'll be covering in this talk-- first of all,
we'll go over what buildpacks are.
We'll talk about why you might want to use buildpacks.
And finally, we'll look at buildpack support
on Google Cloud, how to use buildpacks on Google Cloud,
and we'll do a nice demo tying a few of our products together
using buildpacks.
So just in case you don't want to sit through the entire
presentation, let's do a tl;dr. So buildpacks create containers
without using a dockerfile.
You just give the buildpack your source code,
and the buildpack detects what language you're using
and creates a runnable container that you can run locally
with Docker, or we could run on the Kubernetes platform,
or anywhere else.
So let's just take a look at that in action
on my laptop right now.
OK, so to use buildpacks, we used the pack command.
That is the reference implementation
of a buildpack platform.
And you can see the available commands on pack here.
And we're just going to use the most basic one, which
is pack build.
By default, it builds the current directory.
And we're in the--
you can see-- this java-maven directory here.
And you just need to provide the name of the container image
that you're creating.
And we'll call this hello-buildpacker.
And so you can see it went through a detection phase
where it found a few buildpacks, a Java runtime, Java-Maven and
Java entry point.
It did some analysis.
And it is now building this application.
You can see it's using the Google Java runtime buildpack
and using Java 11.
That's specified by our definition, the Google
buildpacks definition, of what to do when it finds a Java app.
And it is now finishing up its build.
And it has successfully done and created
this hello-buildpacker image.
So now I can just run that.
And we'll bind it to port 80 so we
can load it in our web browser.
This is a server that I have up in GCE.
And that was hello-buildpacker.
And it runs.
And there we go.
It's started.
And now we go over to our web browser.
This is the address of my server.
It wasn't working before.
But if I do a reload--
boom, hello, buildpacker-- it works.
And so you can see, in just a couple of seconds,
we built a Java app and got it running in a Docker container.
And that is a quick demo of buidlpacks.
So that was buildpacks in action.
Now buildpacks allow you to take advantage
of the benefits of containers without needing
to understand them.
So you can isolate your code.
You can create microservices architectures.
You can run on orchestration platforms like Kubernetes
without needing to write dockerfiles or understand
the inner workings of Docker--
so super cool.
Now recently, Google Cloud released a set
of open source buildpacks.
You can find them on GitHub at the address
that you see in this screenshot, googlecloudplatform/buildpacks.
And these cover a bunch of the most popular languages--
Go, JavaScript, Java, PHP, Python, .NET, and Ruby.
And they are designed to create secure container images, which
can run on any of the runtimes from Google Cloud such as Cloud
Run, GKE, or Anthos.
And of course, because they're open source,
you're able to fork them and use them however you'd like.
Now even though they're relatively new,
these buildpacks have powered over 10 million builds
in production.
And that's because they started off as an implementation detail
for App Engine, and Google Cloud Functions.
And they've been powering all of the source builds on those two
products for a while now.
So we're pretty confident in their robustness.
And you can use them confidently too in production.
But in addition to App Engine Cloud Functions,
Google Cloud buildpacks are available on many
of our products.
There's support for them in Cloud Build, Cloud Run, Cloud
Shell, Cloud Code and Skaffold.
And we'll dive into the specifics of that later on.
So that was the overview.
Let's dive in.
What are buildpacks, exactly?
They are a CNCF sandbox project.
Those are newer projects in the cloud-native project.
And you could find them at buildpacks.io.
There are several large companies
contributing to the project in addition to Google.
Pivotal Cloud Foundry is a major backer, as is Salesforce,
via the Heroku division.
And like we said before, buildpacks take your source
code, and they create a container.
But there's a little more to them than this.
And let's take a look at that.
Buildpacks actually run in a container
themselves called a builder.
So here you can see, in a little more detail,
that a container called the builder runs the buildpacks.
And you can see there's actually two scripts that they run.
The first one is a detect script.
And that detects what language your code is.
And the detection, it's actually not rocket science.
It's an ordered list of tests.
And the first test that comes back positive
is declared the language.
So these tests can be very simple, like,
is there a packages.json?
And if that's true, then it's a JavaScript application.
And you can, of course, override those.
But you'll find that they're-- they usually work just fine.
And then after the detection script happens,
a build step happens that actually does the building.
So the builder, it's, itself, an OCI image.
It contains all of the buildpacks, and groups,
and binaries needed to do the building.
And it runs on a platform.
And that could be as simple as the pack CLI command
that you saw in the demo, or something like Cloud Build
or Tekton.
And that actually runs the image.
And you can see here a little CLI snapshot.
The pack command, if you invoke suggest builders,
it'll show you some of the builders that are known to it.
And Google is one of those.
This is also showing Paketo, which
are the-- that's the branding of the buildpakcs from Pivotal
that were released recently.
They're also very good.
And just to add another layer, there's
also a thing called a buildpack group.
buildpacks themselves are very small units of execution.
They can be designed to do just one thing,
like run an NPM install, just installing dependencies.
And a buildpack group is a collection of those.
And that lets you reuse smaller bits of code
like doing a dependency install, and use
those across different buildpack groups.
So when most people say buildpack,
they're actually talking about the buildpack
group that does the full build.
But it depends how the buildpacks are actually
designed.
Finally, there is another piece of the buildpack
called the stack.
And this is the base image, or base layer,
that your application code is put on top of.
Here we show an Ubuntu, stack, which
is what Google Cloud's buildpacks use.
And they can contain some system libraries
on top of that base layer.
And then your application code and dependencies
get put at the top of that layer stack.
Buildpacks cannot do an apt-get to install dependencies.
So those need to be part of the stack.
And we have designed the stacks for the Cloud buildpacks
to contain the most common things that you likely need.
So you shouldn't have to redefine that if you're
using them yourself.
So stringing all of that together, we can see,
here, a flow diagram where the source code gets
taken into the builder image.
The buildpacks and buildpack groups run on it.
They run the detect script to figure out
which language it is.
They run the build script.
They install all of your dependencies, your source code,
whether it's compiled, or it just, direct,
goes on top of there.
And that gets layered on top of the stack.
And then that layer cake that you
see on the right, the green and the gray,
that forms the complete container image, which you
can then run wherever you like.
So that's pretty detailed look into how the buildpack works.
So why would you actually want to use these things?
Well, for starters, a recent study
found that 44% of containers in the wild
had known vulnerabilities for which patches existed.
This just shows that writing your own dockerfile
can actually be quite fraught with errors
unless you're an expert, or maybe
have a platform team that provides dockerfiles for you.
So using buildpacks lets you circumvent that danger,
assuming the buildpacks are designed with security in mind.
Now Google Cloud buildpacks are just that.
They've been security audited.
And they're constantly updated, making sure that they will not
produce vulnerabilities.
Or certainly, at the bottom layer of the stack,
the base layer doesn't have any security issues
that we're aware of.
The containers are built according to our container best
practices--
call that the Google way--
as much as we can via buildpacks.
And all these things let you-- like container novices--
deploy to container platforms like Kubernetes
without needing to write their own dockerfiles.
And even if your organization has adopted Docker,
maybe that adoption is not even, and there
are some teams that haven't yet adopted it,
but they could take advantage of centralized platforms
for running their code.
Another use for buildpacks, even if you do know Docker really
well, is if you just need to do a quick and easy build.
Maybe you're checking out an open source
project or some code that you don't expect
to be using long-term, and you don't
want to write a dockerfile from scratch for it.
So just use a buildpack, build it, try it out, and then, boom,
you're done.
Some specific Google Cloud use cases for buildpacks--
because we're using buildpacks to power
the builds on App Engine or Cloud Functions,
if you want to migrate away from those platforms
to one of our others, like Cloud Run or GKE,
you could take those App Engine and Cloud Function buildpacks,
you could continue to run them against your source code,
and have the same build process happening on your new platform.
And as you know, when you're migrating,
keeping as many things constant as possible
is very, very helpful to minimize the disruption
from that migration.
Finally, another reason to use buildpacks
is because they are so well-integrated into so
many parts of Google Cloud, like Cloud Build,
that you can reuse the knowledge and those tools
across many parts of your cloud platform.
So now let's take a look at support
for buildpacks on Google Cloud.
First of all, we already went over the open source
buildpacks.
You are free to fork those.
Modify them however you'd like.
It's quite easy to do.
And in the README, you'll find documentation
on how to do that.
Cloud Build, which is our builder as a service,
now has native support for buildpacks.
So if you have source code and want it built,
instead of running buildpacks locally,
you can just push that to Cloud Build,
and it will build it for you.
At the moment, that support is CLI only.
But support in the console will be coming soon.
Called Run, our next-generation serverless platform,
supports direct source deployments with buildpacks.
So you can just do a Cloud Run deploy, put your source code,
and it will build and deploy that all in one step.
You can also set up continuous integration
with both Cloud Build and Cloud Run,
where, every time you update your source repository,
it will do a fresh build of that source using
buildplaques-- buildpacks-- and deploy to Cloud Run.
Cloud Shell, which you can access through the Cloud
Console, already has the buildpack tool pack
pre-installed.
So you don't need to do that if you are using our shell.
Like we talked about already, App Engine and Cloud Functions
both are building with buildpacks already.
And finally, Cloud Code lets you deploy to Cloud Run or GKE
using buildpacks as the build mechanism.
And it actually does that via Skaffold, which also now
has native support for buildpacks--
so a really broad swath of support
for buildpacks across Cloud, which
is really pretty exciting.
And now let's take a look at that in action.
And I will show you a soup to nuts built in deployment using
buildpacks on Google Cloud.
Let's take a look at how to use buildpacks with Google Cloud.
So I'm back at my terminal.
And I have a pretty simple Go app right here--
little different from the one we were looking at before.
And if we just take a look at the main HTML file,
you can see it's a pretty boilerplate app.
And all it does once you get past all that CSS is just
pulls some environment variables from the request,
and says hello to you, and displays those.
So let's build it.
But let's start by assuming we actually
haven't installed the pack command,
and just do the entire build in the cloud.
So I'm going to use the gcloud builds submit.
And I'm just going to give it the pack flag
to tell it to build this with a buildpack.
And then I'm going to pass along the name
of the image and the full Container Registry address
that I want to use.
So I'm just going to call this one run.
So we do that.
And we can see it is creating a tar ball
and uploading my app into Google Cloud, into a storage bucket.
And then it is going to initiate that buildpack.
And you can see, right here, it is pulling down the builder.
These are the Docker layers being pulled down.
And it's already cached, so it should be pretty fast.
And so now it's running the detection phase
of the buildpack.
That's where it decides what language it is.
And it should show us that-- yep-- it is using Go.
And then it does the build.
It's going to compile my Go code using Go 1.14.
And it is doing that now.
Looks like it just finished.
And now it's going to create my final Docker image,
putting my application code on top of the stack.
So it's doing that.
And it looks like it is now putting the completed image
into my Container Registry.
And that is all done.
So now that we have a built container image,
let's deploy it.
So to deploy it, I'm going to use Cloud Run.
That is Google's next-generation serverless container engine.
And I'm just going to tell it to deploy a new service to Cloud
Run, and using that image that we just
created from Cloud Build.
So I fire this up.
And it's going to ask me a few questions.
I'm going to choose our fully managed Cloud Run platform.
You can, of course, also use Cloud Run for Anthos
if you have installed that on top of your Anthos cluster.
I haven't, so I'm just going to choose
the fully managed option.
And I'm going to call this my demo-run-for-screencast
service.
That should be unique.
And I'll choose the region.
Let's go US Central.
And allow unauthenticated-- just to make this more interesting,
let's just make this a private service
that you need to be authenticated to use.
So I'm doing that.
And now it is deploying the service.
And Cloud Run services had revisions.
This will be the first revision.
And now its routing traffic to it.
So we're all done.
And it gives me this URL right here, which I
can use to access the service.
And I am going to hit that up.
I'm just going to use the curl command right here so we
stay right in the terminal.
And here's my command.
I'm curling that same URL.
And you can see I'm adding a header with an authorization
field.
And then I'm going to call the gcloud command again
to give it my identity token.
So I do that.
And it's firing it up.
And, boom, we get a bunch of HTML back
that is that container.
You can see, if I do that same thing
but don't pass the authorization token,
then I get a forbidden error.
Because I haven't authenticated properly.
So there you just saw us building some application code
using our service, Cloud Build, with its brand-new native
buildpack integration.
And then it stored that in the Container Registry.
And then we created a brand new Cloud Run
service that was protected via authentication.
And that all took just a couple minutes.
And that is one of the easiest ways
I know to build a new service and deploy it to the cloud
securely.
Thanks very much.
So that was buildpacks in action on Google Cloud.
So in this talk, we have explained what buildpacks are,
why you want to use them, and how
to use them on Google Cloud.
I hope you enjoyed it, and I hope
you'll try buildpacks out soon.
Thanks very much, and have a great day.