[MUSIC PLAYING]
DANNA KELMER: Hello everyone.
Good morning.
My name is Danna.
I'm a software engineer on the Bazel
Team at Google in New York.
Today, we are going to talk about problems
that certain organizations start to encounter
as their company and their projects grow,
in particular with-- in relation to build and test
times, and then go over the ways in which the
build tool Bazel can help alleviate some of those pains.
And finish up with a testimonial from the Angular team
about how it's been working for them.
So why might an organization need Bazel?
Certain problems start to arise as your organization grows.
Your projects start to grow and the needs
of your infrastructure start to suffer with the load
that it requires, and in turn, this might affect your--
the development experience of the people
at your organization.
And in part, this happens because, as you
have more developers, they start to make more changes.
And more changes means that you need to run all of your tests
far more often.
So if you wanted to run all of your tests before every submit,
that starts to increase as you have more people.
The second thing that affects this is as the feature space
starts to grow, this obviously increases just the sheer number
of tests that need to run.
So this is more load that must happen with every change.
And the final part is as your project becomes larger,
each individual integration test requires a ton of compute power
just for a single one.
And so it starts to become, very quickly,
infeasible to run all of your tests on every submit.
So basically, many organizations tend to give up
on integration testing.
And this means that instead, they'll
break up their projects into a number of microservices where
each service is responsible for its own set of tests,
and then those will run on every submit.
And you'll only ever run your substantial integration tests
when you attempt to release.
So this is an example of a way that you
might break up some project.
And what this means is that when you're releasing,
that's the first time that you're
going to encounter problems with the interaction
between these different microservices that
have been working separately.
All right, so what is Bazel?
And how is it going to help this problem
and enable and empower your organization
to run all of these tests as often
as possible on every commit?
So Bazel is a free and open source
automated build and test tool.
This tool has been used at Google for nearly the past 15
years to build all of its source code.
And I'm going to go over the properties of Bazel that
enable this, that enable you to run all of your tests
across your entire organization as often as you'd like.
So the first property is that Bazel is hermetic.
This means that Bazel is aware of all inputs
before the build begins.
So of course, this includes all of your source files
and everything that would need to be built.
But this also includes any tool, for example,
that would need to be used over the course of the build.
And in turn, this hermeticity directly
lends itself to Bazel being incremental.
And so because Bazel is aware of all inputs,
for any change you make, Bazel can very easily
calculate the minimal amount of work
that needs to be done in order to create your finished build
product.
And so, if you make a small change,
you should expect that the amount of work
that Bazel will do will be proportional,
that it will be small.
So I'm going to demonstrate that.
All right.
So here's an example.
We're starting.
We're asking Bazel to build all the tests.
And, just to start, we see that everything is fully cached.
So this is imagining a scenario that you just did a build,
right?
We're starting.
Everything was already built, and now we're
going to make just a small change.
So the change is going to be to a single test file.
So a test-- a test itself, often nothing depends on that test.
So the expectation is that when you make the change
to this single test, that only that single test will have
to be rebuilt and retested.
So here's an example test file.
We're going to add an additional assertion to this test
to make sure that everything, or that this thing is not null.
And then, again, what we're going to do
is we're going to ask Bazel to retest everything.
But the expectation is that Bazel will be smarter,
and it will only rebuild that single test
and rerun that single test.
So Bazel is working and beautiful.
Everything is cached.
But we see it executed one out of 174 tests.
So that's the one test that we edited.
Everything else is still cached.
All right.
The next property of Bazel is that it's deterministic.
So Bazel uses the checksum of all of the inputs
to the build in order to determine whether or not
they've changed.
And what this means then is that Bazel can very safely cache any
intermediary build artifacts that it produces.
And then instead of redoing the work
to do it, if it knows that none of the inputs have changed,
you can safely retrieve the output artifact
from this cache.
And this is a huge opportunity for optimization
at your organization because you're not
bound to keep this cache local.
You can share this crop-- this cache
across your entire organization, and all of your developers
can populate this cache, and not only your developers,
but your CI system can also populate this cache.
And then whenever your developers are doing builds,
Bazel can then check the cache and see
if the artifacts that it needs are already present.
And if yes, simply retrieve them without doing any work at all.
So this is some example of how this cache might be shared.
And of course what this means is that, for example,
the first built of the day will also be fast.
If you say have your cache, and it's--
sorry.
If you have your CI system populating this cache,
then when your developers get to work in the morning,
they do their first build.
That build will probably be trivially fast because almost
all artifacts will be cached, very similar to the example
above before I change the test.
All right.
And this cache can be anywhere.
So you can use Google Hub--
Google Cloud Storage in order to have this cache be in the cloud
not hosted by you.
But you can also set up this cache
to be hosted on premises for everyone to use.
And the final property of Bazel that we're going to go over
is that it's fully cloud ready.
And so because of all the previous properties
that we talked about, its hermeticity
on its determinism, not only can you
store each of the intermediary artifacts,
but you can also do the work on the cloud.
All of the build actions can happen in the cloud.
So we're going to look at an example of another build.
So in this example, in this example,
as before, we're imagining we're starting from a clean slate.
Everything has been cached.
But this time, instead of changing a test file,
this is editing a source file that
is at the very bottom of the project.
Nearly everything in this project
depends on this source file.
And so, now you're going to expect a full cache miss.
You're making a unique change to your project.
And so there's no way that any artifacts will be in the cache.
And so Bazel will do its incremental build,
but the incremental build still is a lot of work.
And so it's going to start working,
but now you're bound by the local compute power
that you have.
So we're actually going to let this
keep running in the meantime.
But as we said, these builds can be moved to the cloud
so that you're no longer going to be
bound by the compute resources that you have.
So, to demonstrate that, we have this project.
And similarly as before, we're going
to start from a clean fully-cached slate,
and then we're going to make the same edit to the same files
that we did before.
And again, this is a file that everything depends on,
so Bazel will have to do all of the work
to rebuild all of the project and rerun all of the tests.
But then when we invoke Bazel in a moment,
we're going to see that we're going
to pass an additional flag that tells Bazel to execute all
of the build actions remotely.
So this is a setup that has been done on this project.
There's a bit of configuration.
And then you pass this additional--
that doesn't work.
Another dash dash config equals remote flag,
and then Bazel knows to run these actions remotely.
And if we see here, we can see that 150 actions
are running simultaneously.
So this project was configured with 150 remote workers,
and Bazel is utilizing all of it.
And this means it's running as many tests
as the configuration allows for.
So now what happens is now that we
see that the number of actions running simultaneously
is dropping, that's simply because there aren't any more
tests left to run.
And so at this point, you're simply
bounded by your slowest running test.
So let's just go back at this point,
142 tests have been executed so far.
We go back to the job that was running locally.
We see that only 20 tests have been executed in the time
that we've been working.
A couple of these tests take a while to run,
so we will move on.
But we see 154 have finished so far,
and they're all running simultaneously.
So 14 are still running.
All right, so that was a demonstration
of how Bazel will run in the cloud.
And now I'm going to pass it on to Alex to talk about Angular
and how they've been using it.
ALEX EAGLE: Thanks, Danna.
[APPLAUSE]
Hi, everybody.
My name is Alex Eagle.
I work on the Angular team at Google.
I'm the lead for our tooling team, which includes things
like making the tools that our Angular users, about 1 and 1/2
million of them, use to develop their applications,
as well as supporting our own team
and our internal development velocity.
And I want to tell you about how much Bazel
has improved our lives.
If you haven't heard of Angular.
It's a enterprise-scale web framework
for a building very modern web applications.
And it is a big project.
So how big is it?
So as Danna pointed out, the development velocity
on your project, it depends on how many people
and how big the tests are.
And you might think that you might--
we need to break up your projects so you don't
run all the tests all the time.
That's really not what we wanted to do.
So we wanted to use Bazel so that we
could run all of our tests all the time
and keep our development velocity high.
And our numbers, I just grabbed these off of GitHub,
and in the last 30 days, we've had
92 authors, over 400 commits touching over 1,000 files.
So this is, I would say, a pretty medium scale.
I know a lot of enterprises have much bigger apps than this,
but it is enough of a scale that we saw
the problem the Bazel solves.
So this is what our continuous integration time looks like.
You can imagine a developer on the Angular team, one
of those 95 people, having to wait an hour for all
of the tests to run, and then they
find out something was broken.
And so you can imagine going through this round trip
many times.
This significantly hurt our development velocity.
And I want to point out here that the numbers I'm
showing here are non incremental, so this is
just like the uncached build.
That's sort of the detail of the fact
that we get contributions from the open source community,
and so using a [INAUDIBLE] cache has some security risks for us.
And so this graph is showing that 60 minutes is
before we enabled the remote build execution,
the dash dash config equals remote that Danna showed.
And today, I just checked this morning,
and a typical round trip time is 7 and 1/2 minutes
to run all of the tests.
So this is a decrease of 87% on our test time,
and it has made a really big difference to the Angular team.
There are other features of Bazel
that have helped us out a lot.
This is easily worth the investment
that we put into migrating our build to Bazel.
So this talk is about developing like Google,
and the Angular project itself is just
a small part of all of the Angular work
that goes on at Google.
We have a lot of customers internally, over 600 of them,
and some of these applications you've probably used.
And all of these applications rely heavily on remote
build execution to allow a very large number of Google
Developers to be productive working in these applications.
And yes, we write full stack integration tests
and expect them to run all the time.
So now that Angular itself has been successfully ported
to Bazel, we want to make it possible for all of you,
if you're using Angular 2.0, to the same thing.
And so as-- we're developing an opt-in preview which
will ship in the next month as part of version 8 of Angular,
that will allow all of our enterprise users
to be able to opt-in to using Bazel.
We will plan to work with them over the next six months
to help them find patterns of porting parts of their builds
to run under Bazel and allow them
to use remote build execution.
And I also want to point out that our plan for version 8
is that, if you don't have a very large scale project,
there's actually no porting required.
Bazel can just drop into your project.
It's really for the enterprise use cases
that some extra migration work will be needed.
So there's one more thing I want to talk about.
We showed how build and test can benefit
a lot from incrementality and can be run by Bazel,
but we're at the cloud conference,
and we are all thinking about deploying services.
So let's talk about deployment for a minute.
A year and a half ago, when the Angular team switched to Bazel,
we also switched how we do our own deployments.
And so, we run basically a loop over our packages of just doing
Bazel deploy of each one.
That's not that interesting because we're a library.
And so our deployment is a shipping
an artifact to a registry.
What's more interesting is that we have an example
application that looks like an enterprise Angular app would,
and as Danna showed, only the change packages
get rebuilt when we do a deploy.
So Bazel's smart enough to know what work it needs to do.
But then, the changes that are produced by that
can go into a new layer in the docker image.
And then only that new layer needs
to be uploaded to the Container Registry because, of course,
Docker has the property that each layer has a checksum,
and if the checksums match, it doesn't need to do any work.
So the incrementality benefits can go all the way from your
build through to your deployment.
And so I have a quick video showing that.
So this is our example application.
And you see there's some text there.
The live edit and deploy.
And so we're going to make a change to that text.
I should point out while this is running
that all the demos you've seen today
were actually Angular itself.
And so those 174 tests Danna was running
was the Angular repository.
That was a demo of our actual day-to-day development
experience.
So now we're going to run deploy dot replace.
So the deploy target is a Kubernetes target.
This is going to go to Kubernetes Engine,
and the replace is just a verb saying what we want to do.
If we were making a new application
we would say create.
And so Bazel, as you can see, 13,000 actions
didn't need to be run.
So we got a lot of cache benefits now.
And then the remaining part, the work
that's still happening here, is we're
bundling up the application.
We're running some slow steps to do the production optimizations
for this application, and it is a medium scale app.
So that takes some time.
And then once the app is fully built,
Bazel knows how to construct the Docker image for it.
And so now we see this image layer is running.
It's running too quickly for me to talk over, which is good.
And then the gzip happened, and so now there's
a kubectl command running.
This is the command line utility that deploys to Kubernetes.
So again, I did all of this with one command,
and I didn't need to say what needed to be built first.
I just said, just like redeploy me.
I don't know what changed.
So now we're going to wait for Kubernetes Engine
to spin up some new workers with the new code.
There's a load balancer in front,
and so now that LiveEdit shows up in the application.
So that's all we have.
We're happy to answer some questions after the talk.
We'll be over here in this area.
Thank you very much.
[APPLAUSE]
[MUSIC PLAYING]