[MUSIC]
Hello everyone, my name is Beka Westberg, and I am a software engineer on the Blockly team.
In this presentation I’ll be talking about Blockly’s shiny new serialization system.
Firstly we’re going to talk about serialization in general.
Then we’ll cover pain points with the existing XML system, and how we solved them.
Then we’ll talk about how to migrate to the new system if you choose.
And finally, we’ll talk about what future work this new system allows us to do.
So, what is serialization?
Serialization is the process of turning the state of the workspace into a file,
and then loading that file back into the workspace again later.
In the old system, Blockly would serialize the blocks, and the variables the user defined.
It would not serialize plugins, or anything else defined by developers.
Now that we know what serialization is, let’s explore some paint points with the old system.
Firstly, the old system was based on XML, which is not very compressible.
You have to save a lot of extra data, which you don’t actually need.
Plus it’s difficult to convert XML to other smaller formats, like binary.
Secondly, the XML format isn’t very easy to modify. It is hard to manipulate programmatically,
while other things (like JSON) can be written much more declaratively
And lastly, the old XML system wasn’t very extensible.
There weren’t many safe ways to add your own data to it, and there was no way to serialize plugins.
So now let’s talk about how we’ve addressed these
pain points with Blockly’s new serialization system.
The new serialization system converts the workspace to JSON, rather than XML,
which addresses all of the pain points outlined above.
Firstly, JSON is more compressible. It doesn’t include as many extraneous
characters. And it is easy to convert to binary formats like CBOR and ProtoBuff.
Depending on the contents of the workspace state,
converting to binary can give you great reductions in save file sizes, especially as compared to XML.
Secondly, JSON is modifiable. It is easy to read manually, and to write programmatically.
And lastly, JSON is extensible. The new format has been structured so
that developers can add their own serialized state, such as the state of their plugins.
Now let’s look at what the API for this new system looks like.
Saving and loading workspaces works very similarly to how it did before.
You use serialization.workspaces.save to save the workspace,
and serialization.workspaces.load to load it.
However, there is one key difference: when you call load() the state of the workspace,
including plugins, is first cleared before new
state is loaded. This ensures that plugins don’t accidentally get into bad states.
Loading individual blocks is again very similar to how it worked before.
You call serialization.blocks.save to save them, and serialization.blocks.append to append them.
Note that when you append an individual block,
the state of the workspace is not cleared. The block is just added to it.
Because the save format has changed, hooks for saving extra state have had to change as well.
For blocks, the old hooks were mutationToDom and domToMutation,
and the new hooks are saveExtraState and loadExtraState respectively
Similarly, the old hooks for fields were toXml and fromXml,
and the new hooks are saveState and loadState.
Unlike the old XML system, the JSON system provides an additional way for you to save state.
You can register a Serializer which takes in the workspace and returns some JSON
representing extra state. This is how the new system allows you to serialize plugins.
So, say you’re interested in solving these pain points, and getting these new features:
how do you access this new system?
So the new JSON system was released in the December 2021 version of Blockly.
This means that if you are a new user of Blockly, you can just use the JSON system.
Additionally, at the same time, the old XML system was iceboxed.
This means that the XML system will continue to exist, and old users of Blockly can still use it,
but it will not get new features or bug fixes.
So, if you are someone who was previously using the XML system,
and wants to use the new JSON system, there are a few changes you’re going to have to make.
There are more details in the migration guide, which you can find linked in the slides,
but I’ll go over the basic steps.
So after updating to December 2021 version of Blockly, the first thing you want to do is upgrade
your block and field definitions. This involves adding the new API hooks that we discussed before.
When you do this, be sure to not to remove the old hooks, because you still want to
be able to load your old XML saves, and the old hooks are necessary to do that.
Then you will want to convert your XML toolbox to a JSON based toolbox.
The migration guide has a helpful script that you
can run in your browser to get the JSON versions of all of your toolbox blocks.
And finally, you may want to look into converting existing saves to JSON.
Depending on your project, you may want to convert all of your saves to JSON in bulk,
or you may want to slowly convert them as users edit them.
To convert the saves in bulk, load up a headless workspace, import the XML and export JSON.
To convert the saves slowly over time,
similarly import peoples XML saves and export only JSON saves.
Now to conclude this talk, I would like to briefly share with you some
of the exciting future work the new JSON serializer opens the door to.
None of these is on our short-term roadmap,
but I am personally excited for the possibility of adding them in the future.
Firstly, moving to JSON means we can provide even more support for making save files smaller.
Schema-less systems like CBOR work out of the box, but
Schema-ed systems like Protocol Buffers are now also an option we could provide in the future.
Secondly, because JSON is more modifiable,
it makes it easier for Blockly to provide a blocks-language versioning system in the future.
This would allow you to deprecate old block types,
and replace instances of them in your save files with new block types.
The new JSON system already includes a languageVersion
key to support adding this functionality in the future.
And finally, the extensibility of the system
allows us to create first-party plugins that persist their state.
For example, this is a necessary part of moving workspace comments to a plugin.
And it would also allow us to create a persistent backpack.
So to summarize everything we’ve talked about…
Blockly now has a JSON based serialization system.
It is compressible, modifiable, and extensible.
And it opens up the door to a lot of exciting future work.
Thanks for watching this talk, and I hope to see you at our upcoming developer summit.
If you have questions you can post them in our Q&A or bring them to Q&A sessions at the summit.