Hi, my name is Andrew, and this presentation will discuss our paper titled:
Volumetric Mixed Reality Telepresence for Real-time Cross-Modality Collaboration.
This work was completed at the University of Melbourne in collaboration with Deakin
University.
The foundation of this work starts with the notion of telepresence.
Telepresence, based on the early concept by Minsky, describes an experience where a person
can undertake actions in a remote location as if physically present in that location.
If we consider this idea towards people, co-presence in a virtual environments sense, is the sensory
experience where a person feels like being there with another person.
To this extent, can a person feel that they are in that remote location alongside their colleague
while remaining in their own personal space?
This work is motivated in exploring this phenomenon.
Capitalizing on the characteristics of supporting viewpoint independence, mixed reality affords
the mutual grounding of communication, where collaborators establish a common understanding
of their partner’s actions and expectations.
There are notable examples of mixed reality telepresence in past literature.
In 2012, volumetric 3D capture with depth cameras described the process to achieve telepresence
through a calibrated array of consumer depth sensors experienced through head-mounted displays.
In 2016, Holoportation produced real-time volumetric avatars captured from multiple
capture pods, where two users in identical remote spaces could be co-present together,
while wearing head-mounted displays.
During the same period, Microsoft released their RoomAlive toolkit, which used Kinect
depth sensors to map people and environments, visualized through a variety of extended reality
technologies.
A common theme in the work we reviewed was the focus on a single visual medium –either
projectors, augmented reality, or virtual reality.
Additionally, we noted the limited number of users involved in evaluations.
Predominately, technology was evaluated in dyads communicating over a given task, limiting
the potential exploration opportunities of telepresence.
From our review, we identified three research gaps which this work will address.
To the best of our knowledge, no prior work has explored the possibilities of a telepresence
environment that incorporates the entire range of the mixed reality continuum; integrating
the physical, augmented and virtual realities simultaneously to create a cross-modal design.
Human experiences of social interaction commonly involve more than two people at a given time.
A rich exploration of telepresence should allow interactions with multiple people.
The impact of fidelity of virtual and augmented representations of avatars and environments
in multi-user telepresence contexts is unclear.
We developed an adaptable real time volumetric telepresence system using off the shelf consumer
depth cameras to explore our human-centred research objectives.
The volumetric capture system is combined with augmented and virtual reality technologies
alongside face-to-face interaction to support multiuser cross modality remote collaboration
scenarios.
Our process incorporates an array of individual Kinect cameras calibrated into a single coordinate
system.
The world-aligned result is either rendered locally or transmitted through our network
protocol to achieve real time rendering to remotely connected clients.
Our platform supports both volumetric avatars as well as virtual avatars to support multiple
embodied techniques.
In this demonstration, you can see an example of our platform from the perspective of two
users.
In this scenario, the VR user is physically located in another room in our building, but
appears virtually co-present with the AR user at the workbench.
On the left hand side, from the view of a remote user in VR, a user is standing at a workbench wearing
a Hololens 2 augmented reality head-mounted display.
The two users are engaged in a conversation about the objects positioned on the table.
You can see the spatial coherence between to the two locations as they use gestures
during their conversational turns.
On the right hand side, you can see the same interaction from the perspective of the AR user.
The remote VR user is represented by a female avatar, with supported hand and arm tracking.
You can notice the avatars right hand change from the neutral palms open to a pointing
gesture.
You will also notice the speech bubble that appears when the avatar is talking, this provides
a visual indicator in multi-user experiences.
We performed a multiuser cross modality experiment to explore telepresence.
The instructional design for this study was adapted from an established training currently
offered by our university’s fabrication lab.
The training introduced the tools and techniques required to safely complete three tasks:
measuring and scribing, sawing, and drilling,
using two types of materials, wood and metal.
The training is designed to illustrate correct tool usage, operation, and sensory feedback
for safe operating procedures when undertaking individual work inside the facility.
The anticipated end users for this style of instruction are students and staff with no
prior training in tools.
The study involved two groups of 4 learners, led by an industry professional.
For both groups, the industry professional was provided with an augmented reality head-mounted
display.
For each group Two learners undertook the training unencumbered
from technology, collocated with the instructor at the workbench.
Two learners remotely connected through virtual reality represented by an embodied avatar.
The learners swapped roles halfway through the training, so each learner experienced
in-person and remote delivery.
The study was conducted across three separate sites.
The local site contained our volumetric capture system configured with 3 cameras around a
workbench.
Using this configuration, three cameras would reconstruct our instructor in 360 degrees
while one camera would additionally reconstruct our two physical learners in 180 degrees.
Remote Site 1 and 2 were located at a different location in our university building connected
over our enterprise network.
Both sites involved a Meta Quest 2 connected to a PC.
The remote site was represented as a classroom, with the volumetric environment and users
positioned in the middle of the room.
Our study design allowed for the observation of key experiences:
With users experiencing telepresence through varying mediums, the design explored the effect
of head-worn displays on the underlying communication.
With the instructor acting as a mediator between local and remote learners, how was their communication
varied between the two representations.
Finally, was there a difference in perception and communication when communicating with
the variations in avatar representations.
Our study included a total of 5.5 hours of video and audio recordings from interviews
and training studies.
We followed a deductive thematic analysis approach outlined by Braun and Clarke on all
of our collected data.
From our analysis, we identified four themes.
Volumetric Experience Group Experiences
Communication Between Modalities, and Mixed-Modality Proxemics
I would recommend viewers to read the paper for more in-depth analysis on our findings.
In summary, with all our participants never experiencing mixed reality telepresence prior
to our study, our results show that volumetric capture produced easily recognisable avatars
that afforded for a natural communication channel.
Our industry professional was capable of seamlessly transitioning their training into the telepresence
setting.
In the following slides, I will give two examples of interest.
Due to asymmetrical nature of the collaboration, there were almost two planes of existence.
One virtual (remote plane) and one physical (local plane).
Our co-presence platform combines these planes altering the proxemics of the combined space.
As you can see in the video, learners arranged themselves at social distances, standing opposite
the instructor.
During the training, as remote users re-positioned themselves, the social distances collapsed
into intimate distances, resulting in feelings of territoriality and awkwardness.
In many ways, the theory of proxemics was validated, with remote participants generally
respecting their physical learner’s space, however with knowledge that their movements
did not directly impact their local colleagues, some learners were more willing to position
themselves in intimate distances to influence their own experience, such as wearing the
skin of a local learner.
Freedom of movement was seen as a valuable characteristic of mixed reality telepresence,
allowing remote users to position themselves in ways unachievable in their physical forms.
From our instructor’s perspective, they were able to focus their attention to both
cohorts of learners, however, due to the asymmetrical nature of the study design, they were conscious
of their efforts to address “unseen” remote learners might have been distracting for local
learners – as they reported “while I am not directly paying attention to local learners, the information
I am saying is important to them.”
This scenario can be seen in the provided video, where the instructor is responding
to a question regarding the drill with a remote learner to the right of the two local learners.
Comparing the insights between learners and instructors on their communication with differing
avatars, volumetric capture avatars were able to convey important body language that assists
in grounding communication.
While remote learners could easily recognise their local counterparts, and respected their
presence, due to their lack of acknowledgement from our study design, they appeared ghostly and unattentive.
This emphasizes the importance of conveying information between both the virtual and physical
planes.
Based on our developed virtual co-presence platform, there are numerous future directions
afforded by the results from this work.
This work only looked at physical training as a mechanism for learning.
Our users expressed the prospect of other domains for learning which would extend our
understanding of multi-user experiences.
The multi-user approach of our evaluation provided unique insights into proxemic interactions
between the physical and virtual planes.
A deeper examination of proxemic interactions afforded by mixed reality telepresence is
needed, especially when the seam between the two planes is reduced.
Finally, our evaluation only considered a small set of novice users.
Running our training over a longer period would provide the opportunity to garner deeper
insights into the interaction, communication, and experience of both instructors and learners.
This has been a presentation on our work: Volumetric Mixed Reality Telepresence for
Real-Time Cross-Modality Collaboration for the CHI conference 2023.
Please refer to our paper for more details and contact us if you have any questions.
Thanks!