Just to get started.
Anyone who hasn't heard of WebAssembly?
Great.
Anyone here who has had a reason to use WebAssembly in a production app?
All right.
So we are all about on the same page.
I would like to invite you back into journey of time to the year 2017.
Like most of you, I heard about WebAssembly but beyond surface level stuff I didn't know
much about it.
I read this headline and thought huh, I guess if WebAssembly really does kill JavaScript
I will figure that stuff out but it seems like people have wanted JavaScript to die
for a while and we are still here.
What does a typical web dev need Wasm for?
I work for a consulting firm based in Seattle called GEN/UI.
I think our dev manager described it best when he said our web expertise is in doing
something weird.
We built a screen recording and editing app for Chrome OS that runs entirely client-side.
Our requirements were as follows: The app was meant to look and feel like it was native
to Chrome OS and act as a competitor to others out there.
The client requested the video provided video editing capability and needed to run locally
offline.
There are a couple ways to develop for Chrome OS and we decided to use Chrome package OS
and they are essentially web apps given additional hardware APIs.
As we started to investigate how we wanted to implement the project, it was a surprise
to discover how much it is possible to do with media in native JavaScript.
For instance, if the user consents, you can capture their whole screen or Webcam and use
the native recorder API to record the screen, create a video and allow the user to download
it to their machine.
That is possible in any modern web browser.
There is nothing special about Chromebook that lets you do it.
If all we needed was a screen recording app we would be set with those tools but the client
had requested some video editing capabilities.
As a result, we started to get into a realm of things that are technically possible in
JavaScript that is just not a great idea for a variety of reasons.
For instance, if you wanted to create a water mark you can create a media stream that collects
data from a canvas element.
You can then draw your watermark on to every frame of that canvas element and merge it
into the stream from your screen recording but unsurprisingly it is too memory intensive.
The resulting videos were kind of like watching a very badly dubbed foreign film.
You have got all the pieces there but they don't line up.
Another video editing feature we wanted to implement was trimming or shortening the video.
It turns out you can actually do that in plain JavaScript so long as you only want to trim
the end of the video.
If you want to trim the beginning, you are out of luck, at least in Chrome.
That brings us to functionality that simply isn't possible in native JavaScript.
Once you create that WebM file you are done.
There is no way to add a watermark or filter after the fact and it isn't possible to re-encode
the video.
The media recorder API gives you a WebM so if you want a gif or MP4 you can't do that
with native functionality.
At this point, we went back to the client and recommended a server-side only solution.
There are two big reasons for this.
Video editing is a memory intensive process and Chromebooks are meant to do most work
from the Cloud by design.
And client side video editing pushes the limits on what is possible today.
We had two people working on this without an infinite budget.
The client asked us to move ahead with a client-side only implementation eventually.
This set us off on the journey of determining what is possibly limiting.
We determined video editing isn't possible in native JavaScript we needed to find a way
to run a code that is not native JavaScript.
Running non-JS code in a browser has a long history.
Anyone remember this thing called Flash?
Anyone?
It is kind of obscure.
Any ActionScript devs out there?
Any OUBT about Silver Light?
You may have heard WebAssembly isn't just another way to write plugins and that is true
but without understanding more about plugins in general it is hard to know why the comparison
is made.
The '90s are in again.
Let's go back to 1995.
We are all wearing JNCOs and the Slater guy is this guy.
Your browser of choice is Netscape and they just released the Netscape plugin API or NP
API which was the first way to develop plugins or browser extension and handle media from
within the browser.
What happened was the browser would come across content it couldn't handle and detect the
file type and load the appropriate plugins that will run in place on your web page.
As of today, NP API is deprecated in every browser.
That is because the Advent of HTML5 means the browser can handle a lot of things natively
that would otherwise have been handled by plugins.
Plugins didn't actually fully die with NP API.
Google released the pepper API which is a build chain that produces binary file types
native client or pepper native.
It was meant to provide a more secure, less complex alternative to NP API.
You can recompile libraries written in C or C++ and that could be run within the browser
and accessed in JavaScript.
For the WebAssembly fans in the crowd this may sound familiar.
Why aren't we talking about pepper and how it harkens the death of the most widely used
program language?
An open standard for pepper was not created and they were only supported on Chrome, Chromium
and blink-base browsers.
Portable native client is deprecated for every platform except Chromebook but we were building
for Chromebooks so let's put a pen in that.
WebAssembly, like portable native client, takes advantage of common hardware capabilities
to deliver native or near native performance and allows libraries written in C, C++ and
rest into a Wasm format.
Developers can write directly in WebAssembly for the super nerds out there.
Unlike pepper WebAssembly is an open standard and has been adopted by every modern browser.
It is really something that you can use and deliver to the vast majority of the web.
That doesn't actually say so much about what is great about WebAssembly in the first place,
or why we would want to use C or C++.
There are a couple reasons why you might use WebAssembly in your application.
The first of these is performance.
Performance on WebAssembly is faster than native JavaScript and that is mostly due to
a lack of dynamic typing in WebAssembly.
You might look into WebAssembly if you have a particularly heavy operation you could perform
in JavaScript but prefer not to.
Every single page web application framework has a DOM diffing operation.
It is a heavy thing to do and it can have negative performance implications for updating
your page particularly in component-heavy scenario.
Glimmer, which is the DOM rendering engine by the Ember team has an existing build that
leverages WebAssembly for this purpose and that is a methodology that should result in
a much faster rendering engine.
I am definitely not saying that because of the primary architecture of Glimmer is speaking
later today.
There are folks looking into this as well for React.
Performance aside, the second reason you may want to use Wasm speaks to video editing.
You cannot edit video in JavaScript but you can edit video in C++.
You can do several things in C++ or Rest like video editing, platform immolation and games.
Wasm was built for our use case and we needed to choose a video editing library we can use.
It was important to find a library because there were only two of us and we had neither
the budget or expertise to build something from scratch.
It turns out there are not tons of video editing options in C++ or Rust and that may be due
to FFmpeg which is a command line tool that supports adding watermark or tripping or cropping
video or anything you might want to do with it.
FFmpeg could be used through portable native client or WebAssembly and we were unable to
find another good candidate.
The first thing we did was go to the mpm registry and there is an FFmpeg.
I am grateful to the author of this package and the person who forked it and added extra
documentation.
Without these folks, we would have been lost but these were enough bread crumbs for us
to figure out how to do this.
Getting back to my MPM install.
I saw a couple concerning things off the bat.
First is the size.
The entirety of FFmpeg is about 15 bytes and that is a lot to load and a lot to ask of
users to download.
The default build includes tons of functionality we didn't plan on using.
It also includes the H2-64 kodak which was still pat patented.
This is an old version, also.
We came to the conclusion there is an existing MPM install for this library, but we needed
to compile our own Wasm file.
This is the point at which we started to get into technologies that JS devs don't tend
to use on a day to day basis and the first is called Emscripten.
For those that haven't done a lot of this thing it is vaguely analogous to a tool like
Babel.
It takes uncompiled C++ library and builds it in a way that a Wasm is produced.
You can pass in various build options.
If you want to build your Wasm binary in a way it is compatible with web workers you
will do that as part of the overall Emscripten and build command.
That is a lot of high level information.
Let's bring it down.
The following example is a little hello, world written in C. Like most of you I don't program
in C# but the main method is what runs when the app loads.
We imported the standard input and output library.
In a C# application this prints to the console and in our app prints to the JavaScript console
but we will display it in the DOM also.
Now we need to compile our hello world to a Wasm file.
You can do that in Emscripten with the following command.
You can see we do need to specify Wasm and that is because Emscripten can compile to
other formats too.
And dash 0 output to HTML lets us know I want Emscripten to build the page rather than just
compile a binary.
Let's look at the web page.
There is a lot more going on than just LEL hello world.
The default gives you a page with a console you can use to see the Wasm output and it
has JavaScript glue code.
If this was a Thanksgiving turkey drawing tutorial we are at this strep drawing an outline
of the hand.
You know what comes next?
We don't need hello, world.
We need FFmpeg and there are a few steps in between.
It is worth just trying here to convert FFmpeg and see what happens next.
Make is a Unix tool.
And clang is a front end compilation for interface and gcc is the GNU/Unix.
This is the root folder of the FFmpeg library.
We need to use the project's make file and substitute cling and gcc with the own Emscripten
build tool.
Piece of cake.
In case you thought it was cake, let me assure you it isn't.
Emscripten built their own tool to help you substitute for clang and gcc.
It is called M con emconfigure.
We will run emmake make and that should build the library into the format and bit code.
Assuming everything is well, we can run a command similar to the hello, world command
from earlier but of course, everything hasn't gone well.
After about 20 minutes of compilation, emmake make has an error generating deep from within
the heart of this massive library.
Anyone who has struggled to get builds to translate knows the pain that can accompany
production builds particularly as they become more complex.
Building a project in C# isn't different and compiling tat project into a format it was
never intended for adds a greater degree of complexity.
Looking at the hello, world.
C from earlier it seems reasonable to debug the process even though we are not C# developers.
It didn't see reasonable to debug this giant make file and dive in the heart of this legacy
framework.
Emconfigure makes assumptions about your C# base build environment that may not be true.
It was necessary to go in and replace every reference to the existing build change.
Emscripten can't use anything using ASM and that is used in any C# libraries to provide
compiler instructions to gcc.
Since we are not building with gcc we need to pass the option to build without this.
And finally, Emscripten will not convert every possible application.
In the end we prevailed.
Here we are looking at a video that has been created in native JavaScript through a screen
recording of a YouTube clip.
Our media recorder returned a blob we can convert to a file and save to the user's local
machine.
You will notice we cannot seek through the video and this is due to a Chrome-specific
bug where their mediator recorder doesn't create key frames as it reports therefore
the find of file created doesn't know important things.
The client wanted to add trim functionality to the app and that makes total sense.
We have started screen recordings for getting to set-up our screen and needing to trim out
the first bit.
This is where FFmpeg comes in handy.
If we run our FFmpeg we will ask it to trim to the desired length.
I compiled my FFmpeg build, create a new web worker and listen to make sure that worker
is ready, once everything is loaded I will post a message containing all the information
that FFmpeg needs to kick off a trim operation.
We can see the type of thing I want to run is called run and that is the default or main
method in FFmpeg and I have passed in data which is actually an array buffer.
Finally, I pass in the same arguments I would pass in if I were using FFmpeg from the command
line.
We have got a dash-SS where you start the video.
Dash T is how long the video is.
Dash c copy means to use the existing kodak.
Let's see what that looks like.
I kicked off the FFmpeg by default in the background and we can see before too long
we get a little video.
It is different than it was before.
It is only two seconds long.
That being said, there are some important caveats for using this technology.
The last video editing feature was to provide crop and that is in case the user wants to
only display a portion of what they have created in their video.
You can see we have kicked off our operation, we are getting lots of logging output from
FFmpeg and that is great.
We know something is happening but that video is only a minute long and we are still watching
it convert so we can't put that in front of the user, obviously.
The other options were really fast?
What is up?
I told you performance in WebAssembly is supposed to be much better, right?
There are two things happening here that cause performance issues.
One is just that re-encoding takes a little more time than copying the exist code act.
This operation would be a little slower from the command line but not by a factor of about
a 1,000 which is what we have here.
It turns out when Emscripten provides the conversion there are a few patterns in C++
that convert but have implications.
Code that relies on X86 behavior.
And yes, I had to look at stuff up there.
In theory, could we address these performance issues and patch them in the same way we watch
the code causing conversion problems?
Or add in WebAssembly specific composition applications to compensate?
The truth is I don't know.
We did haven't the budget to become all-knowing video Wasm encoding masters and who does,
really?
It is worth mentioning when WebAssembly shipped to all browsers it shipped as an MVP.
A ton of functionality is coming down the pipeline that might help support the issues
we had with FFmpeg and one is multi-threaded support.
Wasm is looking to support garbage collection will opens the doors beyond C++ and Rust.
Since we were on the Chromebook platform we looked at using the pepper build change and
an FFmpeg process is defined there and the result is quite a bit faster than our WebAssembly
binary.
Perhaps, unsurprisingly, the performance heavy conversions take time and produce an unacceptable
user experience on the Chromebooks due to their lightweight processor.
Is this a story on why you should not use WebAssembly?
Not at all.
We learned a lot and I want to share those takeaways.
First, FFmpeg was the wrong library to learn about WebAssembly and C++ in the first place.
If you are evaluating a new C++ library you think you mind like to use there are well-known
patterns you can check for to give you a good idea about whether you should proceed.
Portability guidelines can be found on the Emscripten website.
This is where it is good idea to have someone with C++ expertise to determine if your library
contains these and how difficult it is to patch them.
The Rust language was designed with the web in mind and looks to support Wasm.
If the functionality you are looking in exists in C++ and Rust it is probably a good idea
to use Rust.
WebAssembly is helping pave the way to use the browser as a true application platform.
For the average JavaScript converting into Wasm is still a problem.
As Wasm becomes more mainstream, I expect to see better tooling, more libraries and
simpler processes become more prevalent.
In the mean time we may all be consuming WebAssembly in frameworks long before we ever have a reason
to create things ourselves.
Thank you.