Hi everyone, today I would like to present "Causality-preserving Asynchronous Reality"
The authors of this work are Andreas Fender (which is me) and Christian Holz
In the current digital age, we got used to asynchronous communication services like e-mail and instant messengers
The advantage of asynchronous communication over synchronous communication is that the recipient can choose when to process a message or mute notifications to avoid distractions
In contrast, interacting in a physical workspace is always synchronous
While such co-located interactions are important for socializing in workspaces, they can also lead to interruption of co-workers
We explore a future in which immersive technology is commonplace
Asynchronous Reality means that immersed users are able to freeze time around them, for instance when they want to work on a task that requires focus
This creates a "time bubble", which means that time only flows within the personal space of the user
The time in the room is frozen until the user is finished with the task
When time unfreezes, the user quickly catches up with reality
To explore the concept, we implemented a prototype system called "AsyncReality"
We combined a network of Azure Kinect cameras with an Oculus Quest 2 Virtual Reality headset
We also attached a RealSense camera to the headset
Based on our prototype system, we acted out a scenario
Let's take a look at the first part
[Scenario narrator] The scenario takes place in a small game studio
The team is in the middle of their current project
To promote the game, they want to release an action figure that is based on one of their game characters
They already want to start developing the action figure, so that it can be released at the same time as the game
Today, they want to create a 3D printed prototype of it
There is already an animated in-game version of the character
Two team members, Joe and Anna, discuss how to make it suitable for 3D printing
Joe: Right now the model of the character is one connected mesh
They conclude that the model needs to be split into three parts
Joe then returns to his desk and starts creating the 3D print version
After sending the models to Anna, Joe wants to work on the game logic for the rest of the day
So he turns on Focus Mode
To block distractions, noise is played back for Joe and only objects in close proximity are rendered
While Joe is focused on programming, Anna prints the individual pieces one by one
After some time, the first piece finished printing and Anna brings it to Joe's office
Anna: Hey, the skull is done now. I'll just put it here on the table and you can take a look at it when you have time
Let's rewind for a moment
Joe is in Focus Mode, so Anna's message is automatically recorded without him noticing
Our system stores asynchronous events including the recording and changes that occurred in the room -- in this case: the skull on the table
Instead of notifying Joe, the system renders those changes as glossy shapes
At some point, Joe takes a break and turns around
He decides to check the new object on the table
Past Anna: Hey, the skull is done now. I'll just put it here on the table and you can take a look at it when you have time
The object is now up-to-date in Joe's reality and he can freely inspect it and interact with it
Afterwards, Joe resumes to his programming task and returns to Focus Mode
From then on, every hour or so, Anna drops by with another printed piece or some comments
Anna: Hey, the left wing is done now. Let's check whether it fits
Anna: Seems alright. Could have been a bit better maybe, but I'll just put it here for now and you can take a look at it
This second event introduced two changes:
The skull from Event 1 was moved and the left wing was added
[Presenter] Let's stop here and take a closer look
The arrow indicates that the skull from Event 1 is a causality for Event 2
This is because Anna interacts with the skull in Event 2
So this event cannot happen before Event 1, in which she brings the skull to the room
Throughout the scenario, Anna brings more objects and manipulates existing ones while Joe keeps programming
This leads to an instance of a “Causality Graph”, which ensures causally correct playback once Joe leaves Focus Mode and catches up with reality
Let's take a look at the general components of a causality graph
An event contains a recording as well as "Manipulations", which can be something like added or moved objects
There is at least one such manipulation per event
The first type of manipulation is a “Trigger”
These are manipulations that no future events depend on
For instance, if a physical object did not change since the event, then it is stored as a trigger
There can be any number of triggers and it does not matter which one the user approaches to trigger event playback
The other type of manipulation is a “Causality Node”
Those are manipulations, which are also causalities for at least one future event
An event can have any number of causality nodes
Each causality node has a causality for one or more future events
Lastly, an event can depend on any number of causality nodes of previous events
We developed an architecture to turn causality graphs into practice
This overview shows processing steps, data and point cloud renderers
If-conditions within the dataflow are diamond shaped
The primary input of our system consists of the RGB-D streams that capture the room
In the case of our scenario, those streams come from three Azure Kinect cameras that capture the whiteboard, the table and the entrance
We apply some pre-processing to smooth the input and remove noisy edges
The subsequent steps depend on whether the user is in sync with reality or not
Out-of-sync can mean for instance that the user is in Focus Mode or currently catching up with reality
If the user is in sync, we simply render objects and people of the environment live
This means that users can interact synchronously
If the user is out-of-sync, then we use the asynchronous pipeline instead of rendering the point cloud directly
We first subtract the current background, which is a snapshot of the room geometry including objects
Our event detection checks, whether the RGB-D frames differ from that background
If that is the case, we start recording the incoming RGB-D frames
While recording, we also keep track of regions where depth values are close to the background -- for instance when a person touches or moves objects
We call this the "traces" of the event
Once the system does not detect movement for a short while, the recording of the event ends and we add it to the list of events
In addition, the causality inference compares the new event trace with the traces of previous events and updates the causality graph accordingly
Lastly, we take a snapshot of the room right after the event to update the current background
Based on the triggers of the causality graph, we can hide parts of the room from the user that are not up-to-date
We do this by simply masking those regions in the RGB-D stream before passing it to the environment renderer
Using the inverse of this mask, we render the triggers as glossy shapes
Taken together, the masking allows the user to already interact with objects even though not all regions are in-sync yet
The other input of the system consists of the RGB-D streams that capture the area close to the user
In the case of our prototype, those frames come from the RealSense attached to the Oculus as well as the Kinect that points to the user's desk
We always render the point clouds from those streams so that the user can interact with handheld objects for instance
The RGB-D frames are also the primary input for the playback scheduler
Based on the causality graph, the scheduler checks whether the user approaches a trigger and initiates event playback of the requested event plus any causal dependencies
If one object is related to multiple events, then those are played back (one after another)
If all events are played back, the user is in-sync again
This concludes our presentation about Asynchronous Reality
Thank you for your attention