Hi Everyone, My name is Bala and I am a PhD student
at UC Berkeley, advised by Prof. Bjoern Hartmann.
Today, I would be presenting TransceiVR, a work that we did in
collaboration with Cuong Nguyen and Stephen DiVerdi from Adobe Research.
The current status quo for communicating with a VR user, is the
VR mirror. It shows what the VR user sees at that moment. But, the VR user
doesn’t see the external user and their actions, and this
leads to frustrating and inefficient communication between the two.
Such interactions, are commonly referred to as asymmetric
interactions, where, the interfaces used by the two are different.
Before I get to the details, What TransceiVR does is that
it allows the external user to NOT JUST VIEW the VR user,
but also to interact with them, by providing them
With affordances to draw over a 3D space, as
well as share annotated screens with the VR user. These are
rendered as overlays over existing VR applications
To better understand the asymmetrical communications in
VR, we interview experts who carry out these regularly.
From our interviews we found two main problems
First is due to the shaky first person video
feed that is being observed by the external user.
Second issue is broadly about the difficulties relating
to talking about VR scene elements of differing types
we identified that to solve these, one may need to
augment a VR application with additional graphics and interfaces
There are different levels at which one can intervene and augment.
To be broadly useful, we build TransceiVR at the VR platform level.
and it uses interfaces and strategies that can help
augment both open and closed source applications.
Let’s walk through an example scenario, that illustrates working with TransceiVR. The
task is a collaborative 3D modeling task, done
using an existing closed-source application,
named Blocks. Here, A first-time VR user,
collaborates with an experienced external user,
who views the VR user’s activity, in real-time through TransceiVR’s iPad interface.
To onboard the VR user with controllers, the outside expert uses the controller panel.
The interaction is quite simple. They press a button on the panel
And the corresponding button gets highlighted, in blue, on the VR user’s controller.
Note that these blue highlights are rendered by TransceiVR and not the Blocks application.
The external user, then proceeds to explain its role, verbally.
Now, they need the VR user to load a specific 3D model
from a grid of choices. So, they share a prior frame to the VR user,
The VR user places the frame at a convenient location in their environment.
And then, the external user marks the grid element to be chosen.
Using that as a reference, the VR user proceeds to select that 3D model.
Now let’s see how this plays out for the external user.
Since, 3D model grid changes continuously, it becomes
hard for the external user to refer to a specific thing.
So, the external user chooses a prior frame from below.
Then they share that frame to the VR user. They then proceed with
marking the right object for the VR user to pick. This allows the
This allows the external user to carefully inspect dynamic scene elements and
also avoids frequent back-and forth communication regarding these.
After opening up the right model file, the
External expert user wants VR user to pick up a specific part.
They directly annotate the part over the mirrored video feed, and then
TransceiVR projects those annotations at the corresponding position in the 3D scene.
Compared to annotating on a shared screen, This is an
easier and a more direct interaction.
Note that this works, because, unlike before,
concerned objects here are static, and stationary.
If the part is not in the view of the VR user,
External user browses the frames captured at different angles in the VR scene,
selects the desired frame,
and then annotates over that frame. The VR user, is then guided to
that object, through direction arrows, rendered in the VR scene.
This scenario, also resembles our user study task.
With TransceiVR, the users completed the tasks significantly
faster, with fewer errors and perceived a lesser workload.
It is important to note that these annotations, need to be placed at the right depth in the scene,
else, they will not make sense when viewed from a different perspective.
But How do we do this with just 2D video feeds?
For this, We use a pair of video feeds, one each for left and right eye of the VR user.
We compute a live depth map of the scene, through computing
optical flow and subsequently disparity, between them.
To get the right constants for converting optical flow values to depth values,
we calibrate using 2x2 square grids at known 3D positions in the scene. Then,
we fit disparity values of those pixels, with the ground truth depth values
OpenVR’s IVROverlay interface allows us to place
overlays at arbitrary positions in the 3D scene.
From the depth map we know the depth into the scene,
We have access to HMD pose through OpenVR
Using these, we can place the overlay at the
right position and orientation in the scene
For spatially frozen and temporally frozen frames,
we store the corresponding depth maps and HMD poses.
Related work is discussed in detail in the paper.
Besides these, here are some related talks, that will
be presented at this UIST over the next few days.
Limitations and future work include, Scaling the
current interaction design to a group of users
Tuning it, for remote usage And to be able to render 3D objects in the scene
Through this work, we Identified challenges and goals of asymmetric interactions in VR
And we presented TransceiVR system, that
performs an intervention at the VR platform level
to address these issues. Thank you.