- This is a proposal for WebAssembly ES module integration.
It's currently being worked on
in the WebAssembly community group.
This presentation was given in March 2018,
to the WebAssembly community group,
the Node modules working group, and to TC39.
I wanna start with the goal of the proposal.
Today there is an imperative JavaScript API
for creating module instances.
To use a function from the module,
you have to set up the request to fetch it,
and manually set up its imports.
Then you have to manually trigger instantiation.
At which point you can finally call the function.
This proposal adds a declarative API.
In particular, it adds the ES module API.
With this API, the request for the file,
setting up imports, and instantiation,
all happen implicitly.
This should make WebAssembly modules easier to instantiate.
But the bigger benefit is that,
with this, WebAssembly modules will be able
to take part in module graphs with JavaScript modules.
This will make it easy for webdevelopers
to combine functionality from WebAssembly modules
and JS modules.
So from a user perspective
that's why we want ES module support.
From an engine level perspective what do EF modules do?
They provide a lexical environment
for the outer scope of the module.
The way that this lexical environment
is different from function lexical environments
is that the boundaries or semipermeable.
With other lexical scopes in JavaScript,
such as function scopes, you can't have
siblings scopes accessing each other's variables.
But with modules, you can say which variable
should be accessible by other modules
using an export statement and then other modules
can import them.
So this provides a link between them
and makes it so that the importing module can access
the variable from another scope.
ES modules do this in a different way
from other module systems.
It's different from both CommonJS
and from WebaAssembly's JSAPI
which both copy values.
Instead, ES modules use live bindings.
So both the exporting module environment record,
and the importing module environment record
point to the same memory location.
I know this because it has implications
for how WebAssembly ES modules should work.
Before I talk about that though,
I want to do a short walkthrough
of how ES modules work for JavaScript.
With ES modules, you can import and export things.
so we have this whole graph of modules
that are exporting values and references
and other modules which are importing
those values and references.
How does this graph get loaded?
The work happens in three phases:
Construction, instantiation, and evaluation.
Now people talk about ES modules being asynchronous,
and you can think of it as such
because execution yields to the event loop
after each phase.
But the phases themselves are not necessarily asynchronous.
They can be done synchronously.
So let's walk through each step.
First in construction we need
to fetch all the module files and parse them
into module records.
But these aren't instances yet.
We need to have a full understanding of the module graph
before we start instantiating anything.
We start with an entry point to this graph.
For example, the source on the script tag,
and we fetch that file.
Then we parse it to figure out the next set of files
that we need to fetch,
which is the requested modules property on the module record
and we continue fetching and parsing
until all of the requested modules properties are empty.
Half of this work, the loading and the fetching of files
is specified in HTML spec.
The parsing part of this stage
is in the ES module spec.
While we may need to change some language in
the loader spec, we shouldn't need
to make any major changes.
So at the end of this construction stage,
you've downloaded all of the module files
and have parsed them into module records.
Some implementation such a Spider Monkey
or JSC will also create the module environment record here,
and wire out the exports to their memory locations.
But per the spec this isn't necessary until
the end of the instantiate phase.
Some engines also compile the code here.
So after construction we move into instantiation
and two things need to happen here.
If it hasn't already been created,
the module environment record needs to be created here
and have exports wired up,
and imports need to be wired up to those exports.
This happens as a depth-first post order traversal
so we get down to the leaf that has no Imports
and then come back up one level
to hook up the Imports to the exports from that leaf node.
As I mentioned before, these are live bindings.
The important export are pointing
to the same memory address.
In this phase, these memory addresses are filled in
with a temporary, undefined value, not their final values.
We haven't evaluated the code yet
so there are no actual values.
The one exception to this is functions,
which are initialized at this point.
The reason to have live bindings and for instantiation
to be separated from evaluation
is so that we can handle cyclic dependencies.
For example, let's say that you have a cyclic dependency
in CommonJS between A and B.
A exports a value which B imports
using something like de-structuring.
When B imports it, A hasn't yet finished evaluating
so the value was still undefined.
Then we'll return to A, which we'll assign a value
to the export,
but that value will not show up for B.
Because CommonJS handles imports and exports with copying,
the import will still have the value undefined.
ES modules get around this by doing this linking step.
All of the imports are going to be undefined
throughout instantiation.
At this point you're just wiring them up
to the memory addresses.
No values are actually expected yet.
The next step is evaluation, which goes through
and fills in all of the variable values.
It does this by running the top-level code.
There can also be side effects
like requests sent out to servers
so evaluation is only allowed to run once for each module,
and then the instance is cached.
The cache holds on to one instance per module per document
or work or at least for JavaScript ES modules.
These are cached based on their canonicalized URL.
So how would all of this work for WebAssembly?
The first phase is pretty easy:
that's the loading and the parsing.
During this phase we would let
the loader fetch the WASM files,
then when parse module is called on each file,
we would logically construct the WebAssembly module
to get the list of imports to recursively fetch.
Implementations might decide to create
the module environment record
and handle exports at this point.
Then, during the instantiate phase
we would logically construct the WebAssembly instance
from the module given its dependent imports.
After that, during the evaluation phase,
we would run the start function.
That's equivalent to the top-level module code.
An engine wouldn't to have to
actually implement it this way, though.
It could do a light decoding and loading phase
and just pull out the import section,
and then do compile and instantiate
during the instantiation phase,
now that it has all the imports.
So that's how we could add ES module support
in a pretty straightforward way.
What are the problem spots here?
One of the problems spots is live bindings.
They enable cyclic dependencies
and we want to be able to have cyclic dependencies.
To start with, we wanna unable cyclic dependencies
between JS and WASM,
but what does that mean to have
a live binding to a JavaScript value from WebAssembly?
For example if an export and that export
is the numerical value, currently that value's
going to be copied when it's imported.
For now, live bindings will only work
for imports coming from JS into WebAssembly,
if it's a function.
A value won't work because of the time of WebAssembly
instantiation when it reads the live binding,
the value is undefined.
One thing to note: if a web assembly module
exports a function, memory, table, or global,
those exports can be initialized
during the instantiation phase
and they will be available as live bindings to JS.
In the future, if we have both mutable WebAssembly globals
and an any type that could hold any JS value,
then we may be able to have JS live bindings
as mutable any typed WASM globals,
but we'll need to think carefully about this.
Enabling direct WebAssembly
to WebAssembly cycles is trickier.
There's some pretty nuanced reasons why this is the case
which we're going to have to explore more.
So it's unclear what support for WebAssembly
to WebAssembly cyclic dependencies there will be.
This means we're pretty sure we can support
a graph like this with one WebAssembly module
in a cycle and other WebAssembly files in the graph.
But we may not be able to support a graph like this
where two WebAssembly modules depend on each other.
On the topic of cyclic imports, currently the ES module spec
disallows cyclic imports for any module
which is not a JavaScript module.
We have a good idea of what needs to happen here
so we just need to work with TC39
to change that wording in the spec.
Those are examples of some of the problems
we're going to need to work on for ES module native support,
but we also need to support bundlers.
For those who aren't familiar,
bundlers are what have been enabling web developers
to use modules in the browser
before native module support landed.
But even though native support is here in the browsers,
bundlers are still likely to be used
by a lot of projects because they reduce
the number of HTTP requests that you have to make.
They do this by combining bunches of modules
into a single file and adding
a little bit of a run time to load them.
They're probably going to continue to rely on the JSAPI
at least until you can have multi-module files
but they want to emulate the ES module API
because that's what their users are authoring
and those users expect it to have the same semantics.
With the current JSAPI that we have for WebAssembly modules
it's difficult for bundlers to emulate ES modules
while still using instantiate streaming.
This is because instantiate streaming requires
that you pass in the fully initialized import object.
However, even if you're only importing functions
this is only available after the dependencies instantiate.
This means that you have to sequentialize things.
One possible way around this is to allow
the properties of the imports object to be promises.
So those are the kinds of issues
that we'll need to figure out as part of this work.
So why should we start now?
The bundlers are already actively adding support as is Node
and we need to be in a position to give them guidance,
otherwise we could end up in a situation where de facto
semantics are set in place by
the ecosystem outside of this group.
So what are the steps to move forward on this?
This has been accepted as a phase zero proposal
in the WebAssembly community group.
We'll be working with that community group
to push this through the rest of the phases.
We'll also be working with TC39
and the Node modules working group,
and we have implementers from the major browsers
and Node who are interested
in validating this work as we go.
If you want to follow this work,
you can check out the WebAssembly ESM integration repo
on GitHub.