Automated Performance Budgeting Into Your Pipeline With Sitespeed.io
- Daniel Lopez
Hi, thank you.
Mic check.
Hello.
All right.
So the official title for this is automated performance budgeting into your pipeline with
Sitespeed.io.
It's kind of a mouthful and also I put W there for some horrible reason, apologies for that.
and I normally don't do public speaking, I'm very happy that we have the HPE track for
those who are beginners, or yeah, and I just really wanted to share this tool, because
I think it's awesome.
I think it really doesn't get enough credit, and it's been around for an extremely long
time, and I just wanted to share it with you.
So that's what I'll be doing with you here today.
So a quick overview, so we're going to talk about my own personal performance journey,
some -- I guess just like performance budgets in general, what they are, what they're for,
and we're going to talk about Sitespeed.io specifically, and performance budgets within
Sitespeed.io specifically.
We're going to talk about, there's a lot of different tools that do performance budgeting,
Sitespeed.io is one of them and there's a lot of other really amazing tools that I found
with in my journey.
So yeah, we'll be talking about other tools and things to consider, and I wanted to throw
out some stuff that I've been experimenting with, and what I see as my future for front-end
performance, what I'd like to see, and what I've experimented with.
So first: A little bit about me.
There's me behind my pizza curtains, that I sewed myself and put up in my office and
some things that I like.
I go by the name of debug xyz on the interweb and that's because I have way too many hobbies
for my own good.
I'm not good at any of them.
So this list here is all things I'm really bad at.
So I'm into growing Pequin peppers, tech minimalism, is hard.
Accessible is hard.
Skateboarding was really hard.
turntablism is insane, I'm horrible, horrible at that one, but I love learning about it.
Front-end engineering is insanely hard.
Right now I'm learning NixOS which is an operating system that is configuration as code.
I've been doing, I don't even want to say how long I've been taking front-end seriously
but I'm relearning CSS and investing a lot of time in that, as well.
So I'm going to quickly go through two of my hobbies, don't worry, I'm not going to
go through my whole list of hobbies.
One of my brothers gave me chilly Pequin seeds.
He suggested I give it a shot.
They have a 15% germination rate.
Normally birds only eat them and they kind of eat away the hull of the seed so you kind
of have to emulate that to make sure that they're happy and keep them O
Thrasher, so this is a '90s thing.
I'm see if I can resize my slide here.
There we go.
Yeah.
Here we have the pose of the year, so at this point I'd been skateboarding for a very long
time.
I had -- it was an absolute obsession of mine.
It takes tons of practice, and is -- it's difficult.
And so when I was skateboarding, I would go all around downtown, I'd grind the -- you
know, destroy the curbs at my local library, and it would get really hot and the (speaks
Spanish) when I would get really hot and exhausted from skateboarding, I would go into my local
library, and cool off, and play with a computer.
And that's where I actually learned to code.
And I would print out -- actually JavaScript, even though I gave up really quickly, and
take it home for studying, and I did that, because it was hard.
So when I knew I was going to be a programmer, I decided I'd go to boot camp.
That was hard.
So this is me at my graduation photo Parris Island and there's my father and mother supporting
me at my graduation.
It was hard.
Frontend is hard.
It was very difficult and achieving web performance can be very difficult.
It's a discipline -- like front-end web performance is a discipline within front-end and maintaining
web performance is even more difficult.
You can really work really hard, you know, getting your site to get that time to first
interactive super-low, but at the pace that we jump from job to job, and from project
to project, you're going to have a lot of people from with varying skill levels, and
everyone within front-end performance has their specialties.
There's amazing front-end developers who are super knowledgeable about accessibility, but
may not know about front-end performance, so it's important to share your skillset with
your peers and make sure that you're having front-end discipline meetings to kind of share
your expertise within the field.
So you have to do all of the things for front-end performance: Capture metrics, observe trends,
provide alerting, fixing actual issues on the website, because that's what, you know,
-- what we're talking about here is that's the kind of context that I come in from is
the web, and front-end development.
So fixing issues on the website, raising awareness with your peers.
You know, you may go through this same struggle if you're working on things like security,
maintainability, maybe you're super into testing, you've got to be sure that you're raising
awareness within your peers, as well as consulting, tools, analysis, for analysis, providing -- supporting
platforms.
One of the first things I did when I started focusing front-end performance within my organization
is I just started capturing PSI data because I thought that was really important for reasons
that I'll talk about later.
And just all of the things you have to do.
And it's -- and it's pretty intense, you know, especially if you're a smaller shop, and you
may have all of this within your organization, but you still need to provide that consulting
to your peers to make sure they're aware of the tooling and of the structure that's there.
Performance is critical on the web.
I'm not going to talk too much about this, but CPU are getting really fast.
We have blazing fast mobile devices, but you know, not everyone has that, and most of the
cell phone adoption right now is happening with like new people buying cell phones who
haven't had cell phones before, are buying low to medium-tier devices, so it's super
important going into the future.
And as the web advances, expectations are getting higher.
People want snappier experiences.
Users gret frustrated, they tend to get less likely to come back and nobody likes a slow
site.
This is us today.
I actually like I think that was funny and stuff, but I like I literally do this.
I literally, you know, do this to my salad, like it's really gross.
[laughter]
So yeah, most of you are probably concerned about profit on your websites.
This is it.
So Step 1, Step 2, performance, profit.
Check it out.
No, seriously, there's amazing resources by Google and there's this book that I read called
time is money, the business value of web performance that if you're trying to sell web performance
to your organization, you gotta read this book.
It's amazing.
It gave me tons of case studies to show to my leadership.
And, you know, you know, you could -- don't do an A/B test for it.
Look to other studies, seriously, because it's very difficult.
I don't recommend it, And you're ignoring a key part of the equation.
There's a lot to it, it's very complex, but, you know, SEO is a big factor to revenue,
so where you're, you know, with Google having tools like Lighthouse and PHP insights being
directly factored into your page rank score, this is one of the key pieces of advice that
you know, Google lets you know is a big impact.
So I forgot some things.
In that big list of all the things and that's regression.
So it's -- it's easy to accidentally, you know, you have a project where you need to
convert a timezone, so you pull in a moment and then you realized that you need moment
timezone plugin and then that pulls in literally all of the geographical latitude and longitude
information about the timezone lines that you didn't know about, and wasn't documented.
So it can be an easy mistake to make, and you need tools to support you throughout that.
And also, it's about scalability, as well, it's kind of a weird term to use in this sense,
but I think about organizational scalability.
You can't be there for everything as, you know, maybe that one or maybe there's a couple
performance experts and you have this large organization, or a large project, so I tend
to speak within the context of organizations.
But you may be working on an open source project where performance is absolutely critical.
It may be for Mobile Web.
And I'm still probably forgetting tons of things.
So what are performance budgets?
It is what it sounds like.
You define something like a budget.json, you can specify some things.
Typically you might want to failure build.
It's a bit of a touchy subject to roll out to your organization, but I'll talk about
that a little bit later.
How can performance budgets help?
First of all, it gets the conversation started.
When you introduce performance budgets into your organization or project, you're immediately
going to start talking about performance, which is immediate when?
But what's more important, and one of the keys to performance budgeting is it keeps
the conversation going, so this is, you know, when you're introducing a new feature and
then you roll it out and then it fails to build, you're like, oh, well, lesson learned,
I'm going to make sure that I bring up this conversation a little bit early, because it's
kind of pushed our project a little bit out.
You're making explicit tradeoffs.
Instead of implicitly kind of making that tradeoff, you're being direct and more -- you're
being more direct about it.
There has been a conversation, there has been a commit, maybe you even bump your budget.
That's OK.
It works like that in real life.
If you've ever tried budgeting and like going on a hard core budget, the first thing you
do is overrestrict yourself.
And you adjust, and that's OK.
But it's being explicit about it that kind of helps out with the situation and it prevents
accidents.
This is one of the biggest things for me.
So there's a lot of different things that you can budget on.
So there's score-based metrics.
If you're using something like Lighthouse you might be familiar with, it gives you an
overall performance score, so you can assert on that.
So if your SEO is critical to your organization and you need that traffic coming in to make
your revenue, you know, that score is going to be important, so your SEO team might even
demand that and then there's quantitative.
So this is about numbers.
It's about how many kilobytes of JavaScript do we think we really need?
And kind of asserting against that.
It can be how many images you have on your home page?
And maybe it will remind you that oh, I need to lazy load.
That's one of the things that, like, I do as consulting within my organization.
Lazy load.
Lazy load, lazy load.
So timings, as well.
So timings can be difficult.
In fact, this is one of the things that I really wanted to touch on, because when you
roll out performance budgets into your organization, the last thing you want is false positives,
especially at the beginning, so that's one of the key advices -- pieces of advice that
I have, is to make sure that if you're budgeting, make sure that your timings or that what you're
budgeting on doesn't have false positives and that your timings are stable.
Before I forget, you know, because this is an important part, I have some advice on keeping
your timing stable, so within the type of websites that I work on, there's lots of third
parties, and that can cause a lot of volatility in your metrics, so you don't want to include
that within the metrics that you're budgeting on.
It is important that you keep an eye on those things, but make it separate and budget for
them separately.
Accessibility budgets!
This is about performance budgeting, but it's an easy thing to do, so don't forget to check
the accessibility box within your performance budgeting tool.
A lot of them come with that so you can assert against that, and it's an easy win for accessibility.
Just a caveat here: Automated accessibility detection only catches like an extremely small
amount of actual accessibility issues.
I've seen percentages as low as like 3%.
But that doesn't mean that you can't take advantage of some of those easy wins of making
some simple mistakes.
So don't forget to check the check box and don't forget to use your label tags.
So back to performance: How do you roll out performance budgets within your organization?
I mentioned that it can be easy to be too aggressive, and it can be difficult to sell
it.
One of my main pieces of advice is when you -- when it comes to rolling out in your organization,
is to don't immediately jump to the slowest item on your page.
Go to the fastest.
Because really this is about regression and keeping a conversation going, and I'd recommend
if you have a fast area already, do that first.
Let's see here.
No, you really have to get everyone on board, so that's actually why I'm here.
This is my sprint demo.
So yeah, you have -- you know, I'm going to be giving this presentation to my organization,
as well.
Another thing, you know, make sure you plant those seeds early, so I've been talking about
this for a long time within my organization, and people are asking me for it now.
And that's a wonderful thing.
Some places to focus, places, you know, areas that are really fast, new projects can be
a great place to start.
Just for like my context, the happy path is a great place to focus.
This is the areas of your website that the users are going to be kind of jumping through.
Some ideas of things that you can budget on, so things like time to interactive, speed
index is an amazing visual metric.
And I was exposed to that by another insanely web tool, web page tests.
People look at it all weird, but it's really an incredible project.
So just wanted to mention that tool.
Yeah, bundle size, basically anything within the performance API, so like, sitespeed,io
specifically, it has incredible metrics.
We'll talk about it more.
Sitespeed.io is very pluggable.
It's been that way for a while, and yeah, you can, you know, I want this checkout button
to display within this amount of seconds.
It's pretty awesome.
How do you figure out what your budget is?
At Google I/O 2018 I think they released a performance budget.
It's a slider to kind of estimate how fast you want your page to load.
And then slide like how many kilobytes of JavaScript and whatnot.
So that's an amazing introduction of a nice little tool.
Analyze your competitors.
You know, you can actually -- there's a little SEO trick to find your competitors.
I actually don't remember.
Yeah, you know, just look at your competitors, see how they're doing.
Another thing is that when you're working against competitors, you know, just a slight
difference between your competitor doesn't really make that huge of an impact.
You got to be way faster than your competitor to actually notice.
There's statistics on this, and studies on this.
What happens when you make a mistake?
What happens when your budget fails in your pipeline?
You can, you know, obviously optimize the feature that you're introducing.
You can remove other features, you can redesign the feature, you may even catch this early
as you're, you know, talking about it early in the design development process, but it
can just as easily be redesigned afterwards.
Or just don't do the feature at all.
It's an option.
Less is more, seriously.
It's so easy to forget.
So here's a screenshot of Sitespeed.io budgeting results.
I don't think I'm going to have enough time to do an end-up demo, but the innovation within
Jenkins is pretty awesome.
The documentation is incredible on that.
And yeah, we'll talk a little bit Mr. Sitespeed.io.
It has something called Coach.
This is the thing that gives you advice.
It has a philosophy of trying not to give you any bad advice.
If you follow the Coach, you will win.
Browsertime is the thing that kind of actually opens up the web page and tries to capture
measurements.
It's actually very cool technically if you deep dive into it.
There's the thing called XVXB it's like an X display server, which is super-cool.
It uses a neat protocol.
And to dig into Sitespeed.io even more, it has this tool called pageXray, shout out to
the designer these are all little icons in the project.
There's this PageXray.
It's used on the compare.sitespeed.io which is a waterfall comparison tool.
It's amazing work.
I just love this project.
And, yeah, I definitely want to deep dive into that at some point.
Here's an example of some of the format here.
This is where if you wanted to write a deep dive you would write your own information.
The compare.sitespeed.io, I'm obsessed with waterfall comparisons, I'll have to jump through
some of these slides because I want to talk about the future here, especially when this
comes to water fall comparisons.
I really love Sitespeed.io because it fell within the technologies that I use at work.
So we use Graphite and Graphana, you can directly import to have insanely nice displays of all
of your metrics.
If you're using Sitespeed.io not just for budgeting, but for tracking metrics separately.
The you know, there's this cool thing called Throttle.
I haven't deep dived into it, but there's some really interesting stuff about this.
A lot of the volatility.
Again, volatility within your timing metrics is critical, there's this kind of network
replay thing that is experimental right now, that it would capture what, during the task
what the network activity was, and it would replay that activity, which is a pretty insane
idea.
And Throttle is inspired by the funniest GitHub project name I've ever heard in my life.
So we're talking about a tool that slows down your website.
It's called Comcast.
[laughter]
It's just incredible.
I jokingly say read the manual, because the documentation is insanely good.
There's tons of integration, instructions for -- I implemented it using it my own personal
GitLab instance and it wasn't too bad.
I didn't read the manual, don't recommend not reading the manual.
I kind of did it on my own, and yeah, and I also integrated work on Jenkins instance.
These are documented integrations for GitHub Action, Jenkins, Travis, CircleCI, GitLab.
The concepts are basically the same across platforms, but difficulty varies.
For me, setting it up on Jenkins was difficult, because at work, we use just a plug for our
blog, that our DevOps engineers are amazing, so engineering.rei.com, they write a lot about
infrastructure as code there, so I had to deep dive into Jenkins DSL, which is, you
know, a little subset of groovy, in order to implement this pipeline as infrastructure
as code.
So it seemed hard, so I gave it a shot.
And you know, I don't expect you to be able to read this and I wish I could pull it up
a little bit more, but yeah, this is what Jenkins DSL looks like.
I specified what I recommend if you're rolling this out to a large organization, is to provide
some default, sane, and generous budgets and so that's what I tried to do, but you want
to have that flexibility so your different teams who have different requirements and
different needs can override those budgets.
So that's what I did here.
So I pulled in some default budgets, merged it with the project budget, and then did an
assertion which basically just ran Sitespeed.io as a Docker image.
And then I let -- we use Bitbucket now about that, and the Jenkins integration is awesome
because it uses the HTML publisher plugin, which Sitespeed.io will output like a full-blown
website with a video, showing the page load, illustrating the visual metrics and when your
build fails, you know, you can output some text that will link you to a nice little,
you know, whole suite of HTML that you can kind of view. and kind of figure out what's
going on with your budget.
GitLab should be pretty straightforward, follow the docs.
There's lots of other budgeting tools and lots of amazing, absolutely incredible tools.
So I've -- one thing is that WebPack actually has performance budgeting built in, you just
may not know it so if you want to roll out some sort of basic performance budgeting,
into your organization there's a flag within WebPack that you can flip to make sure that
it fails if your bundle size is over some insane amount.
So you can make that assertion there.
Lighthouse is obviously amazing.
It has Sitespeed.io actually has Lighthouse integration, so you can use Sitespeed.io to
run Lighthouse and assert against Lighthouse scores as well as other Lighthouse metrics.
But Lighthouse is a great place to get started if you at all care about SEO, which a lot
of people do.
Speedcurve, I've experimented with this, it's incredible, as well.
And the future!
So I think regression analysis is kind of empty.
Maybe I just don't know much about the toolsets here.
But I've done some basic experimentation.
We've set up a private webpagetest instance, within our organization that we can run tests
against and I just ran a job that ran webpagetest private instance on real mobile devices and
started plotting that information out on a graph using D3 and what I haven't seen done
before, and this may exist is you know, you see the graph go up, but OK, what happened?
Why?
So what we did, you'd select a low point of the item that was ran by webpagetest and you
select a high point and that took you to the wonderful waterfall comparison tool where
I've caught seriously so many issues while we have this experimental prototype up.
It's, you know, the ease of being able to observe regressions is important.
Within my own personal journey, setting up webpagetest is not straightforward.
And especially with physical devices, and I think it could be easier if tools like webpagetest
and other tools could start using the cloud.
So there's a lot of cloud services that are providing device forms, and if these tools
can start integrating with those device forms, instead of plugging in your real phones, you
can start using services like G cloud or alias Device Farm to connect and run your tests,
but right now these are mostly focused around actual mobile app tests, so I don't really,
you know, I don't really see a straightforward way to kind of run those tests.
In the browser. and integrated in the tools.
I've spent a ton of time kind of doing research.
Here's a couple of links, and the Google I/O talk is basically the same as this.
I highly recommend checking that out.
There's a Google has put out so many amazing tools and guides and guidance, so I'll actually
publish these slides and put even more links and credits here.
That's my talk.
Thank you everyone.
I appreciate it.
[applause]