Hello, everyone! My name is Lei Zhang. I am a PhD student from the University of Michigan School of Information.
Today I’m very excited to be here to present our paper "FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Vritual Reality"
This work was done with my awesome adviser, Steve Oney.
In recent years, Virtual Reality has captured the popular imagination in movies,
and has been a new medium for content creation.
While the immersive experiences and intuitive interactions are vivid for end-users, designing and building VR applications can be difficult and frustrating because it requires knowledge of not only 3D modeling, but also computer graphics, imperative programming, and reactive programming.
Even Visual Programming Languages (VPLs) built for non-experts, such as Unreal Blueprints, require that users mentally translate between 2D and 3D representations and predict how their code will execute in VR.
Therefore, our main question is: What if the users can create, edit, and program the virtual scene while immersed in VR?
This paradigm is called immersive authoring.
Here is a film named World Builder from 10 years ago, where the main character has built a virtual world with natural interactions when immersed in the virtual world.
Ten years later, we have made progress in immersive authoring in both industry and academia.
However, a common limitation of existing immersive authoring tools is that they are not expressive enough to define reactive behaviors, a fundamental requirement for most practical VR applications.
For example, commercial tools such as Google Tilt Brush allows users to sketch a rocket, but it is not expressive enough to enable users to, say, define when to ignite the rocket, the trajectory of the rocket, etc.
Reactive Behaviors are behaviors where the application responds to events like user actions, system events, or collisions. These behaviors are what make the VR scene interactive.
Therefore, we make the following contributions in our paper.
Our system is based on the concept of FRP.
Let’s use a simple example to illustrate the mechanism of our system. Let’s say we are rockstars and want to do a rehearsal in VR. So the target behavior is to let the spotlight follow the user and switch the spotlight on/off using VR controllers.
So first we have a light as a model and an avatar as a data source. Each model or data source will have a list of attributes such as position and light direction and a list of methods such as lightOn and lightOff, which indicate the actions that an object can take.
To do that, we first introduce a simple vector calculation of direction B to A equals position A minus position B
So here we introduce a subtract, which act as an operator that takes two inputs on the left and produces the computational result as output on the right.
So we first draw an edge from the position of the avatar to the + input of the subtract operator
Then we specify the minus input of the operator
Then we pass the output to the light_direction.
We can also make the light switch reactive using another data source - the controller.
Now besides the light-following effect, you can control the light switch in real-time in VR. And the stage is yours.
We implement this idea in VR, enabling users to compose such flow-based diagrams in edit mode
There are many more example applications that users can create using FlowMatic, demonstrating its expressiveness.
With the time remaining, I just want to note that FlowMatic also introduces mechanisms of taking abstraction of nodes, mechanisms of type visualization and constraints.
We also conducted a comparative study with A-FRAME, a desktop authoring method.
If you are interested, please see our paper and demo video for more details.