We Started Using Webpack and it Took a While - Salem Hilal
>> Hello?
Hello, hi!, oh, my God, oh, my God, oh, my God, I'm so excited!
My coworker, Salem, is about to come up on stage and tell you about the migration process
of moving to Webpack and what a kind of disaster that was.
But it's a really interesting disaster, I promise.
So Salem gave me some fun facts but I'm going to go totally on my own and tell you my fun
facts and favorite things about Salem.
The first thing that you need to know about Salem is he gives the world's best hugs, literally,
so all of you need to be good-enough friends with him by the end of the conference so that
he could give you a hug and trust me, you will, like, love it for the rest of your life.
The other thing that you need to know about Salem is that we have regular Etsy-wide smash
tournaments, we had one the other night.
I was not there, but my husband went, and apparently do not let him play at duck hunt
because he will hide in the corner and throw beans at everybody.
All right, so everyone, gave it up for Salem Hilal.
[applause]
Oh, my gosh, that was the nicest thing that anyone has ever said about me.
It's also very bright up here, everyone's right, everybody said that already.
Thank you, Katie, I'll say thank you so much to everybody who has organized JSConf, it
is unreal to be up on stage.
I can hear my voice echo.
That makes it sound like I'm in a stadium, that's so cool.
My name is Salem, if you know about the cat, and my pronouns are he/him, and I work at
Etsy and we're hiring.
This is my very first conference talk, as well, so if you have any feedback for me,
please do not withhold it.
I would love to hear about it.
So.
OK, cool I wanted to talk about how we migrated from an in-house system to Webpack and specifically
I want to talk about why it took us so long to do that.
I work at a team at Etsy called the web platform team.
If you've ever worked at a place that has a front-end infrastructure team it's probably
very similar.
This talk is essentially about how we moved from one build system another, but this doesn't
mean I'm going to talk about -- my goal here is to talk about the weird problems that we
unearthed and how we fixed them and where all my hair went.
[laughter]
So this talk is like about five parts.
I'm going to first tell you about a little bit about Etsy and our old build system that
we called Builda, and then this really weird rolling it out.
So that's going to be a fun one, so if the beginning parts are boring, at least stay
till the end.
So before I get into it, there are two caveats that are worth mentioning, number one, the
decisions that we made were true at the time we made them.
JavaScript as you all probably know changes very quickly, which is the whole reason that
we wanted a really flexible build system in the first place and No. 2, all of the things
that I'm going to talk about today are big team efforts.
I know a whole lot about how Etsy's front end infrastructure on behalf of my whole team
that kind of collectively built and researched all the things that I'm going to talk about
today.
They kick ass, and I really love working with them.
I think Joe is right up there somewhere in the front.
That's my teammate Joe.
There's also John and Natalia who are not here but they're great to work with.
It's hard to tell you about a migration process without giving you a little bit of context.
So here it is, this is Etsy's codebase, it's like anyone's codebase at a medium sized company.
Some.
That is to say, the code that makes Etsy show up in your web browser from the API all the
way down to the CSS lives in one.
We call our mono repoetsy web, it's worked surprisingly well for us, it has helped us
facilitate.
One thing that did not scale very well when we moved was the JavaScript part of our codebase.
And.
So this system is something that we named Builda.
For those of you who may not be familiar with JavaScript build systems, a build system is
something that takes our files, resolves their dependencies, bundles them together, makes
them as small and efficient as possible, and (audio is breaking out).
we got to hit play on this part.
It's going to be like a carnival game where I have to aim on the screen.
Boop?
One more time.
Yay, there it goes, so when BBuilda was first written, it wasn't this sleek interactive
home page that you see here, complete with cool animations and a responsive navigation
bar and a really cool branding.
It looked more like that.
Having one file depend on another was pretty infrequent and when that did happen it was
usually for large globally scoped libraries, like JQuery ...
Production builds took very little time.
At Etsy, this is pretty important.
We deploy code dozens of times in a single day and slowing down deployment is something
we'd try to avoid at all costs.
So developing JavaScript was similarly pretty simple.
That's supposed to be me.
If we load a page in development that requests some JavaScript, that JavaScript file would
be built from our source and sent back inside of one request.
To keep track of all the possible files that we might need to request, we kept a list of
all of them which we brilliantly called the build list.
When we served a file we'd make sure it was in that list somewhere.
The build list was used in production then later to determine the entirety of what we
needed to build when we deployed [inaudible] in development if a requested file was not
the build list we'd respond with some JavaScript that threw us an exception.
This ensured that we could essentially build any file we wanted on demand.
We used require.js to manage our delinquencies.
Maybe you haven't even heard of this because it's a relatively old library but we still
used it and it was pretty simple up until we moved to Webpack, that is.
We'd follow it down with some of our own code.
Require.js then allows our own code to use any style module definitions as you can see
here.
For some of you this might sound out of the stone age but for us it was pretty cutting
edge when we did this.
Async ... We defined an array of imports which require
then resolve for us.
All of our code goes to the body of one of these callbacks where it has access to all
of the imports.
Builda's job was to make sure that all of the necessary modules were bundled into one
JavaScript file so that require.js has access to everything it needed in the browser.
At some point, React came along.
React is just a library that makes writing large, client-side applications a lot easier.
Sometime around 2016, we decided it would be a really good fit for some of our seller
tools.
Everyone was excited.
React, however, strongly relies on a syntax that looks like HTML controversially aligned
into JavaScript.
If you want to be able to use this syntax, we need to make a build system that it can
do more than just build files together.
So we used an old version of.
(Audio breaking out) So it's a stapler.
I added the staples last night.
There was one problem with this setup, however, because we built every file only when it was
requested rather than when some part of it was edited we had no way of knowing if it
had changed or not since the last time it was requested.
So we had to rebuild every asset every time we wanted to use it.
Transpiling JSX into JavaScript -- so React heavily ended up taking a good bit longer
in order to be served.
But JSX was new, so few few people felt the pain of long rebuild times in development.
Besides, it's just JavaScript, right?
We know it didn't last very long.
All of these quirks together meant that loading something large, like a single-page app implied
rebuilding the entire thing from scratch every time the page reloaded, plus our codebase
was only getting bigger, we had over 1,000 separate JavaScript assets to output.
React code was use in a lot more places and it started to take the better part of a minute
to show up on the page in development which made iterating very, very tedious, on top
of this, developers were starting to ask for the ability to use ES6.
It felt like a nonstarter for us, React support was difficult to implement and when it worked,
it certainly wasn't sustainable.
We knew we needed to do something.
This is where I've told myself that I'm going to drink water and you all are going to take
15 seconds to introduce yourself to the people next to you so that you don't hear me gulping
over the microphone.
All right, time's up.
I'm going to test all of you later so I hope you all paid attention.
OK, I gotta hurry up because I'm already running behind my schedule.
(mic is cutting in and out.)
Can you hear me OK?
>> Can you still hear me OK?
All right, awesome.
OK, cool.
So I drank my water and it is now time for me to talk about the solution to every one
of our problems.
People started talking around this time about this brand new open source build system, it
was called Webpack.
You might have heard of it.
These are all the GIFs for my entire presentation, they are just right here.
All right, any new developer that we hired, asked if we could use Webpack and the internet
made it sound like every tech company that used JavaScript also used Webpack.
It sounded a lot like Builda in that it built stuff, but it also sounded very different
in that it had a robust development experience, it was highly extensible and it had a bunch
of built-in performance optimizations for our code.
Most of all it was supposed to be very fast, especially for development.
So we did a little research.
I wanted to point out, this is a fast chicken.
[laughter]
It's a recurring motif, so keep it in mind.
So we did a little bit of research, we evaluated a bunch of alternatives and we decided that
going down the path of building our JavaScript with Webpack instead of Builda was the right
path.
We had two requirements for any new system that we wanted to adopt.
Number one, it would have to take care of development builds without too much interacts
from developers, one of the nicest things about Builda, you could navigate the entire
site without having to worry about how the Builda requirement looked.
Etsy deploys code to production dozens of times in a day.
If our builds took longer than five minutes, Etsy's deployments would slow down, which
is not a good thing for us.
So Webpack, what was cool about Webpack for us, well, it's highly configurable.
Instead of a build list, Webpack uses a configuration file usually called Webpack.config.
From how you'd like to support templates, if any, to what sort of performance optimizations
you'd like to make.
This oftentimes means really large really scary config files, which isn't always the
right idea for small projects.
But having something that could adapt and fit to our code was very invaluable.
We spent a while writing out a really nice Webpack.config file and also different features
that Builda currently supported which at the time included everything from transpiling
templates.
After spending hours, setting things up just right, we finally got Webpack to build our
whole codebase, but it wasn't quite what we expected.
It took about a half hour to do.
It ate up 20 gigs of our servers' memory and maxed out every single one of its cores.
This is a real screenshot.
And it had the unclear progress indicator that everyone has come to know and love.
[laughter]
This made it exceptionally hard to figure out why it was so slow.
So we definitely had a lot to learn about improving Webpack's performance but at the
end of the day we needed to change a lot about how our codebase worked and how Webpack worked
with it if we wanted to use it with Etsy.
All right, I'm drinking water again, so tell the person next to you what your text editor
of choice is without trying to convince them that yours is the right one.
[laughter]
OK, how we doing?
Good?
All right.
I gotta hurry up.
I am, like I said, not on schedule.
I hope the answer was vim, but that's OK if it wasn't.
[applause]
OK, our first order of business was to get a development workflow that was good.
If engineers could develop with Webpack and validate that their code still worked with
Builda in production we wouldn't be able to offer new features like ES6 yet since Builda
didn't know anything about ES6 transformation.
But maybe we could let the engineers have a good time doing it.
And so why does Webpack take so long to build.
When Webpack first starts it does a complete build of all of your code first.
Then it enters watch mode.
It kicks off a portion rebuild every time it sees one.
This pattern developed to be extremely fast.
We figured that the longer builds would probably be much better but if that initial build took
a half hour and ate up all of our memory, it wouldn't be all that good.
So our codebase was really big and was getting even bigger.
So it would have to scale.
Well, Webpack makes inferences about your code in the context of your whole project.
For example, it is smart enough to group commonly requested modules together so that they can
be cast efficiently between pages.
It can only make these optimizations if it understands your entire project, not just
your individual files.
This is good if your project is nicely scoped but it makes a lot less sense in a mono repo.
So we started by trying to make Webpack faster and less resource intensive in general, something
that's really hard to do both of at the same time.
We got a lot of mileage from caching plugins like cache-loader and hard source and allowing
modules in parallel but at the end of the day we still had a prohibitively heavy first
build time.
I can't believe I found this image.
It's amazing.
We also looked at Webpack's development server.
The development server chooses to keep everything in memory.
For us this meant that our whole codebase, plus all the transformation information about
the whole codebase, plus all of the build files for the whole codebase had to fit in
memory.
There isn't a way to turn this off without modifying the development servers source code
which is actually a really nicely written repository and if you're interested in how
these things work, I would totally recommend reading through the source.
Even if there was a way to turn this off, it wasn't clear that it would save ourselves
from the size of our mono repo over time.
One really easy solution was just to make a bunch of different config files for different
areas of the codebase and only build the one that was being used at the time.
We can make each region be small enough so that memory is reasonable and initial build
times are closer to one minute rather than 11.
For us this meant splitting our codebase into 11 regions was
the right number.
Juggling a bunch of configurations just to browser on Etsy on your work machine was not
ideal.
This meant that we probably needed to write something of our own.
So we wrote a thing called Kevin.
Kevin is a plugin for the express web server framework.
And its only job is to manage a bunch of Webpack instances.
Why did we name it Kevin?
Look how funny he is.
When a request for any file comes in, Kevin determines when Webpack config is responsible
for building that file, if any, and then it starts a Webpack instance for that config.
Once the compiler is finished, it starts running that code in watch mode rebuilding files as
they're edited.
If Kevin gets a request for another file in a different config, it does the same thing.
If there are too many compilers running, Kevin will shut down that compiler based on a frequency
and recency model.
This keeps us from building the whole codebase and it has an added benefit of making sure
that the codebase has regions of it that are all related to each other.
The only extra thing to consider is that really long first build.
If a request for a JavaScript file times out after 30 seconds, and starting up the Webpack
server takes closer to a minute we're going to end up timing in development for what seems
like no reason.
Kid in time out.
That's a joke.
It is modeled after the Domino's Pizza tracker.
We also exposed some data about the status of every active compiler so that the overlay
is able to monitor the status of its build.
Keeps memory usage low and keeps engineers informed about the state of all of their builds
without requiring them to manage a build system.
With any luck, Kevin will be available as open source software very soon.
All right, another water break.
In 15 seconds or less, tell a neighbor what you did for adventure day.
Go!
All right, that is the 15-second warning.
Let me really quickly talk about productionizing your code and then that GC bug that I've been
waiting for.
I want to talk about localization, since there's a lot of other less interesting things that
went into getting our code production ready.
So at this point development was working.
We even started to onboard a handful of developers so they could start giving us feedback, most
of which was pretty good.
Because we weren't building production code with Builda packet.
The benefits of Webpack brought to the development speed were wins in and of themselves.
Our next order of business was to get production to have that same speed.
There is that chicken again, reminding us that we need to get five-minute production
goals.
A working development environment meant that we were at least able to successfully build
our code.
This involved minifying it but more critically it meant localizing all of our assets into
11 different languages.
Localization is a little tricky with Webpack.
You need to kick off a separate Webpack build.
Webpack's localization plugin uses a file that provides localized strings that you provide
and.
It seems simple enough.
Using this method, constant strings can be defined in one file and swapped out between
builds without your JavaScript changing.
There is an implication here that is a little tricky.
Webpack ends up with a different configuration because the plugin needs to be configured
differently will for each language.
This means that in order to run 11 different languages, you need to run 11 Webpack builds.
Again, 34 course, 62 gigs of ram?
It's really big.
We could maybe run two in parallel and have the resources and get like 8 or 9 minutes
for those two builds.
But that would still mean that all of our builds would take like an hour.
We couldn't just ask for new hardware, let alone
a dozen or so of the most powerful computers that we used to have.
If we were trying to keep pace with Builda, we were going to have to cheat a little bit.
Luckily for us, cheating is OK, because Builda cheats, too.
Let me explain what I mean here.
Builda doesn't build that code 11 different times.
Later, when that Builda's mostly done and it looks like word salad, it goes back through
each file, finds those placeholders and swaps them out for actual localized strings in every
language.
If you speak French, you will know that this is an accurate translation.
Compared with Webpack's apparently deprecated international plugin, this seemed like a much
faster approach.
I actually didn't know this was deprecated at the time I wrote this talk, so like I said,
JavaScript changes very quickly.
Anyways, we tweaked our plugins and had them insert placeholders, instead.
From there, we would do a find and replace.
This is actually a lot easier than it sounds.
This whole screenshot is essentially all of the code but here are the juicy bits.
We're able to take our source code and call dot split on it with our placeholder.
Would iterate through the odd indexes and translate them.
In case you're curious we get our translations from a separate service that we maintain.
This message was actually so fast that we were able to provide localization in development.
That's why Kevin has Italian on his forehead.
We could do this quick enough by translating things as they were requested, rather than
build every file into every language possible.
Thanks to this method we were able to get our production builds ready in well under
5 minutes.
OK, in 15 seconds or less, tell your neighbor if you can speak more than one language.
OK, that is about 15 seconds.
Give or take.
I heard more than yes or no, so I don't know what you were talking about.
For the last part of this talk, let me tell you about that 4 milliseconds that kept us
from building this whole thing 3 months earlier than we actually did.
So we felt we were ready to try Webpack on code production traffic.
We were very excited.
A bunch of our end to end tests showed that things were the same.
However, build systems are very hard to replace.
A new build system has to work in every language in every page in the site like we talked about,
but it also has to work on every browser, no mart how old or cutting edge it may be.
So what do we want to do?
We run an A/B test.
A stock photo of an A/B test.
We actually ran five separate ones, and they all look pretty good.
You can see the important metrics that we checked.
And yet, in spite of all this, we had some pretty alarming changes in every one of our
browser performance metrics.
Every page was running way slower than we expected.
Some of our pages are up to 14% slower which is unheard of.
In the general view if your site's performance gets worse, so does your site's money.
We had to figure out what was going on before we could launch Webpack.
Before I go on, let me talk about our client-side metrics very quickly.
Many of you probably know this sort of thing, but some of you may not and I hope this is
a good refresher.
At Etsy we track a lot of things, but two in particular.
One is DOMContentLoaded and one is page load.
So when the browser parses our HTML it goes through line by line and evals everything
one line at a time.
It downloads and executes the whole thing before continuing to parse the page.
Some things like this image tag will -- here is our JavaScript at the very bottom.
Like CSS, JavaScript is downloaded and executed before the browser can continue.
Some JavaScript code, like a network request or timeout calls do not block the browser
and are added to a list of tasks that the browser can take care of at a later time.
Finally it fires the DOM-content loaded event.
There it is!
Once all of our subresources load -- that is a thing that you can buy on Etsy.com
Once all of our subresources load like that image and when the browser has run enough
tasks to really take care of it says that the fires the page.
Both of these were slower for us, this raised two questions: What is so small, yet so different
that we've miss it had?
And two, if something is so different, why do people still buy things on Etsy.com so
we started investigating or performance monitoring code and we double-checked that we weren't
shipping extra code to our clients but both of these things turned up nothing so we decided
to investigate how our code was running in the browser.
This is a flame graph for the listing page that shows the item for sale on Etsy.
The graph shows our code built with Builda.
Horizontal axis represents time the every rectangle represents a function call, and
everything below that is a subtask related to it.
This shows what the browser does when it parses the main JavaScript file on the page.
Now, what does Webpack's look like?
A lot worse.
This is the same file as the previous graph, but it's taking 4 times as long.
For kicks, here they are side by side.
Clearly Builda is doing a lot less.
So this is kind of concerning.
So why is this happening?
Even if our conversion numbers look good, this big difference may mean that we're missing
a performance win.
Let's take another look at the whole graph built with Builda.
This is roughly the area that we were looking at before.
This is where DOMContentLoading fires, this is page load and this is where our JavaScript
file was actually getting executed.
So zooming in a little bit, that rectangle there is the main body of our JavaScript file.
In other words, our performance metrics were not accounting for the time it took to actually
run our JavaScript, but rather when that JavaScript was built with Builda.
Code built with Webpack did not have this problem.
So we no why people were still buying stuff, the same things have happened, but our measurements
have moved.
So let's look back at our two metrics, specifically at load, and very specifically let's look
at the part that say there are no outstanding scripts waiting to be executed.
What if the browser was being tricked into ready.
Very quick because I know I'm way over time.
As ooh reminder, requires a library that Builda was built around.
We sent a copy of it to our browsers, where it's responsible for starting up our code.
Here's where require.js kicks off our execution and it's wrapped in this next tick function.
It looks like it's just a 4ms set time outscore.
This isn't a hard and fast time and the browser is totally allowed to push things back a little
bit later if it needs.
In other words, require.js tells the browser that none of our code needs to run right away.
Require.js may do this because it gives the browser a second to finish loading the page
before starting any JavaScript.
It also might do this because set timeout was a great way to make your performance numbers
look good in 2010.
So knowing that this was a bug, what did we do?
We did not wrap all of our code in set timeout calls.
That would make our numbers look better but it wouldn't tell us anything that we needed
to D but we did add some new metrics.
We added a timer to every one of our top five pages.
Here yellow is Webpack and lower is better.
Those timers were supposed to measure how long it took for all of our JavaScript to
run on those pages.
We were very happy with this graph.
OK, I'm about done.
In conclusion it took us almost a year and a half to move to Webpack.
That is a very long time and it's a lot of hair-pulling.
At the end of the day, we had a lot to learn about build systems, performance and our own
codebase.
I felt like there should be a three-point slide thing, but there were so many winnings
that we had that it's kind of hard to pick 3.
But I tried anyways.
Number one, your codebase gets bigger, not smaller.
If you're trying to find small performance win, it probably won't last very long.
Number 2: Always question what other companies, websites, articles and blogs claim to be best
practices.
What works well for one person may not always work well everywhere.
If we accepted that Webpack's internationalisation plugin was the only option we'd still be waiting
for our JavaScript to build.
And finally if you can take a risk on a big build project, you absolutely should.
We really hated that that metrics bug cost us months of time but we ended up learning
so much more about performance, the browser and our code and at the end of the day after
almost a year and a half of wild problems and pulling my hair out, Webpack is pretty
great software.
So that is my talk.
If you have any questions, please hit me up on Slack or Twitter.
If you like this stuff, my team is hiring and of course, thank you all so very much.
[applause]