Video has become ubiquitous on the web,
and it's become quite easy for everyone
to record and edit and distribute high-quality video.
But video editing today is also a quite solitary task.
It's performed by individuals using single-user applications
on personal computers.
And only a few video editing tools
support collaboration of any kind.
Also, there's a gap between the simple tools
that casual users use and the sophisticated tools
that experts use.
We have apps like iMovie on the iPhone for the casual user,
and applications like Premiere Pro here for experts.
And the simple ones are easy to use but offer
limited functionality, and the complex ones are costly
and they have a steep learning curve,
and there's quite a gap between these two.
Also, tools for video editing typically
do not support working with both live, recorded,
and procedurally generated video.
Video editing is trapped in applications.
We have applications for casual users,
applications for experts, applications
for editing recorded video, and applications
for editing live video.
In our work, we are interested in how
we can move beyond this application-centric software
paradigm for video editing.
And one of the things that we've been exploring
is the notion of information substrates.
And information substrates are software artifacts
that are neither applications nor documents
but can act as either depending on the use situation.
These information substrates, or substrates for short,
they are malleable.
They can be changed and reprogrammed.
And they are composable with other substrates and shareable
among users across devices.
And these substrates can be manipulated
through tools that are embedded in themselves
or through other substrates.
So what would that mean for video?
Well, that would mean that we would have a substrate that
could handle recorded and live video that would allow
us to edit and express time-based properties of video
and handle multiple video clips.
It would embody mechanisms for composing video spatially
and composing video with other graphical content,
but also allow us to procedurally generate
video and manipulate video through programming.
With a video substrate, interaction
would happen through interchangeable and composable
tools rather than applications.
And interactions would be collaborative between users,
potentially users using different tools
and also distributed across multiple devices.
But let's try to make that a bit more concrete.
So Videostrates is a prototype of a video substrate
that we've created to show the potentials
of a substrates-based approach to video manipulation.
And it's also a toolkit for creating
real-time collaborative video editing tools.
And a Videostrate is, in essence,
a web page that represents a video composition.
The URL of this web page can be sent to a service
and then be rendered as a video stream.
Here we have a simple Videostrate
shown to the right where I've opened up the document to see
the HTML, and the same video being rendered in a Videostrate
player to the left.
And this is just a 15-second video
consisting of three clips from our accompanying video.
Videostrates uses a simple declarative notation
where video elements with the composited class
will be included in the video based on a given start and end
time.
Because a video now is just a web page,
it also means that we can use CSS to, for instance,
create spatial compositions of clips like I've done here
where I've just basically composed the clips from before.
We can also use CSS animations.
And notice here that I will now change the document
to the right by adding an animation I just
pasted in here.
And you'll see that it immediately has effects
over to the left, and now we have these wipe-down animations
when the videos appear.
So in Videostrates, any HTML element with a composited class
define parts of the video and can
have a basic attribute specifying
when it should start/stop in the overall video, the offset
into the source material, and the speed with which
the clip should be played.
Then we can add CSS styles to spatially compose videos,
but also to add animations from a set
of predefined animations such as fade-ins of clips, for example.
Videostrates are stored in Webstrates.
And Webstrates is a web-based platform
where changes to pages on the Webstrates server
are made persistent and synchronized
to all clients of the same page.
And you can see that here where I have the same Webstrate
open in two web browsers and the changes
synchronize between them.
In the example before, the Videostrate
was stored as a Webstrate on a Webstrates server
together with another Webstrate that
represents the video player.
And the video player asks the Videostrate's streamer back
end to create a video stream of the given Videostrate
and then control the playback.
And any change happening to the Videostrate
while it's being streamed will live update,
even while the video is playing.
And Webstrate has this simple API
for playback of Videostrates where
it's possible to ask the server to stream a specific URL,
play, pause, and seek, also record the playback,
but also render the Videostrate, frame by frame,
to a video file, and finally, also
broadcast the stream to an RTMP-based service
such as Twitch, for instance.
With this very brief introduction
to the basics of Videostrates, let
me try to show you some of the things that this enables.
The first example I will show you
is collaborative video editing.
And for this example, we've created
this simple timeline-based video editor
inspired by Apple iMovie.
And this video editor is itself a Webstrate
that then transcludes, as it's called, another Videostrate
in an IFrame.
And the Videostrate then consists
of references to source material that, in the editor,
is shown in the top left.
The timeline is shown in the bottom,
and then the top right is the preview of the video.
So now consider this scenario.
We have Alice and Bob here.
They're interaction designers that
are creating a video mock-up of an interface they
are working on.
They record some clips on their phones.
Then they open a link to the Videostrate
on their phone, which allowed them to upload the videos.
They appear in the video editor.
And now they can collaboratively start to cut the video.
Bob here opens up a preview on a large display
on another computer that live updates as they cut the video.
What Bob does now is that he opens up
a different tool, a subtitle editor, that
points to the same Videostrate and starts adding subtitles.
The timeline editor doesn't know anything about subtitles,
but they still appear in the preview of the video.
And finally, they want to add some effects,
but the timeline is too primitive.
So they just open up the developer tools in the browser
and edit the Videostrate directly to add in an effect
like you saw me doing previously.
So what did we just see?
Well, what you saw here was that a Videostrate
enables real-time collaborative editing, even between two
different independent editors, in this case, a timeline editor
and a subtitle editor.
And we see how Videostrates can enable cross-device interaction
here simply by opening the URL of the Videostrate
on a mobile device that then provides a form for uploading
video clips.
We also see how decoupling video editing primitives
from applications allows just using the developer
tools of the browser when you hit
the ceiling of an application.
The editor uses the video streamer API
to create a preview, and it also uses a cache
on the streamer to generate thumbnails and fast seek
preview videos.
The subtitles are simply DOM elements
that are added programmatically to the Videostrate and styles
using CSS and composited into the video.
All right, the second example is programmable video.
And here we have Grace.
She runs a small one-woman video production and effects company
and wants to create a small opening logo for her videos.
She records a bit of video on a green screen
and then imports that video into a combined
Videostrate and computational notebook that she's
made that uses a shader to do chroma keying to replace
the background with an image.
She now renders this to a file that she can export and then
import into another Videostrate notebook
that she's been working on.
And here, she overlays the video as a texture in a 3D scene.
And while she tinkers with the code,
she can see the live updates in the preview to the left.
So what you see here was that a complex web
page such as a computational notebook
can be rendered as a Videostrate if it just contains elements
that Videostrates understands.
And the notebook here, in this example,
is a Codestrate, which is a notebook system that we've
built on top of Webstrates.
To programmatically manipulate and generate video,
we're using what's called custom elements in Videostrates that
allows, for instance, to draw on a canvas when
the time of the timeline in the video updates.
And the example to the right just
shows a simple example of how you
can connect a function to when the time updates to draw shapes
on the canvas.
My third and final example shows how live streaming can
be handled with Videostrates.
This time, the scenario is a comic book convention
where Beth, a volunteer, is setting up
some cameras for live streaming.
And she has an interface implemented as a Videostrate
for selecting which feeds to show on the convention web
page.
And this also includes ad hoc feeds
from mobile phones of other volunteers.
And Beth can record these composed streams
to a file for archival.
And you'll see here, in the bottom right,
is what Videostrates sees and records,
so that's recorded from Videostrates.
And then she can also live annotate the video
by opening up the Videostrate on her iPad
and drawing on top of the video feed.
And finally, she hooks this whole thing up to Twitch
to save some bandwidth on the convention web page.
So what did you see here?
Well, in this example, we're using Videostrates
to produce a live stream that can
be shared using a platform like Twitch
and recorded to a file for archival.
We use Webstrates' in-built support
for creating WebRTC video streams between clients,
and then just basic CSS and DOM manipulations
to compose the different streams.
The Videostrates streamer opens this Videostrate and then
simply records what's shown in the browser, which then happens
to include some WebRTC-based video
streams and some annotations in SVT
and a transparent background.
Now note that, in this Videostrate,
it's not time-based.
There are no clips with a start and an end.
Here, we're just using the Videostrates platform
to record and stream a composition
of live and annotated video streams.
So how does Videostrates work?
When a client requests the Videostrate server to stream
a Videostrate, the GStreamer controller
on the Videostrates server instantiates an instance
of Firefox in a browser's sandbox.
And this sandbox has a frame grabber
that grabs all output from Firefox to the GPU.
And Firefox now opens a dedicated Webstrate
that includes the code for controlling playback.
And this Webstrate then transcludes the given
Videostrate that we want to render
and injects JavaScript and CSS into it for playback.
The GStreamer controller now returns a WebRTC stream back
to the client.
And if we look at the client side,
for example, the timeline-based video editor,
this is itself implemented as a Webstrate that
transcludes the Videostrate being edited
and then has a reference to a client-side GStreamer library
for controlling playback.
And this library handles both playback
and also caching of media files for a preview.
Notice that the client also transcludes the decompositor
Webstrate.
And this is done because all of the playback control
happens through a WebSocket-like mechanism in which Webstrates
called signaling over this particular Webstrate.
Some implementation details I want to mention-- more
can be found in the paper.
The Firefox instances, they run in security sandboxes
using what's called Firejail where the frame grabber is
injected into a dynamic library.
We use the GStreamer media handling framework
for everything related to WebRTC.
And we do not actually play videos.
But instead, we completely control the playback
using the seek-to-next frame method
on video elements that, in Firefox, is
extremely performant.
And then, for CSS animations, they
are synchronized to the time by pausing them and then just
simply changing their delay.
It's hard to say something general about performance
as it depends on the complexity of the Videostrate.
But on our admittedly powerful consumer-grade computer
we use for it, we can have simultaneous playback
of six Videostrates with 4K source material
to a 1280-times-720 output screen.
The current limitation of--
Videostrates has a number of limitations.
First of all, we do not support audio at all.
This is future work.
And we do not have any error corrections on the WebRTC
to see stream, which means that it's quite sensible to network
latency.
But Chrome now just ports forward
error correction on WebRTC streams,
and we did some tests last week that seem quite promising.
It's also very easy to do a denial of service attack
on Videostrates because you can just render a Videostrate that
has an embedded while-through loop in it,
and then the server dies.
We're also facing some limitations of the browser,
for instance, how it's not possible to render a DOM
subtree to a WebGL texture, for instance.
So in summary, what Videostrates does
is that it moves video editing capabilities from applications
down into the medium.
And Videostrates is also a toolkit for creating
real-time collaborative video editing tools where
you can mix using recorded, live, and procedurally
generated video.
And we've shown a couple of examples of use,
but we believe that Videostrates opens a new design
space for web-based collaborative tools working
with video that we hope the community will explore with us.
And with that, I would like to say thanks for listening.
[APPLAUSE]
Any questions?
Yeah.
Go ahead.
Please, next speaker, please set up.
[INAUDIBLE] from the University of Toronto.
Nice talk.
And as I more like social computing and about
live streaming, I wonder how your tool can actually
enable more--
especially in live streaming actually, viewers
want to interact more with other viewers
and also with streamers.
So do you think your system can enable
new ways of interaction between viewers and streamers,
for example?
Yes, potentially yes.
So for instance, when we stream using Twitch,
it's a bit of a one-way street.
But in principle, the viewers can also
access the Webstrate that's being streamed.
And there could be a particular view on that for the viewers.
Of course, there are scalability issues.
So when you stream on Twitch, you might have 100,000 users,
and the Webstrates platform doesn't support
that many concurrent users.
But on a smaller scale, that would definitely be possible.
Any other questions?
Yeah.
Go ahead.
Hi.
[INAUDIBLE] from University of Rochester.
Great talk.
I wonder how hard it is to learn to use this tool.
Is it targeting developers?
I wonder how novice users will pick up
this tool to create your [INAUDIBLE] also
showed in your talk.
Right.
So obviously, doing video compositions by hand
requires skills in HTML and CSS.
But the idea for us is that we can actually
build video editing tools on top of it.
And in principle, it's limitless what kind of tools
you can build on top.
And then you can imagine building tools
that are easy or more difficult or can be extended by the user
through installing new components in it or what not.
But just for a regular web developer,
it's very, very easy to express a video composition.
I would say that someone that has basic HTML skills
would find it easier to make a complicated video
composition with Videostrates than with a regular video
editor.
Of course, that's just a claim, but--
OK then.
Let's thank the speaker and then let's move on to the next talk.
[APPLAUSE]