When you're talking in video conferences, have you ever caught yourself pointing
with your hands on the screen as if others can see where you are pointing?
Sadly, this kind of body language is not possible in current video-conferencing systems.
In our work, we built a prototype of a malleable video-conferencing system to explore new
mechanisms for deictic gestures and study their implications for inclusion in hybrid meetings.
My name is Jens Emil Grønbæk and I am a postdoc at Aarhus University.
This work is done together with my colleagues Banu, Carla, and Clemens.
When communicating through video, our body
language is reduced to these small "talking heads"
and we are missing a shared frame of reference for pointing,
because What You See Is Not What I See!
This is especially bad for deictic expressions like "here" and "there". We often need gestures
to support these expressions, such as pointing with the index finger
to an object or person in context. These are called deictic gestures.
In this work, we focus on hybrid meetings, where multiple co-located participants
communicate with remote participants through video.
These kinds of meetings often suffer from primary room dominance,
where remote participants tend to feel excluded from the group of co-located participants.
To address these problems, we built MirrorBlender.
MirrorBlender is a prototype designed to support deictic gestures
in video conferences with conventional hardware.
In conventional video meetings, people share video feeds of themselves and content from
screen mirroring. And together, we call these video feeds "Mirrors".
"MirrorBlender" then supports mixing and blending
these mirrors in any way desirable for the meeting.
So how does MirrorBlender support deictic gestures?
Remember, using your real body for pointing on the physical display in the room does not work.
In order to point on screen content so that everyone can see it,
you must adopt the illusion of using your own representation in the camera mirror for pointing.
We built the prototype according to three principles.
1. WYSIWIS
MirrorBlender establishes a shared frame of reference
where everyone in the meeting sees the same synchronized 2D view.
2. Malleable Mirrors
Every user can manipulate every mirror by moving
them around and changing their size and translucency in the WYSIWIS interface.
3. Mirror Blending
Mirrors can be overlapped and made translucent to create a blended image.
This principle supports integration of interpersonal space and workspace,
and it's building on the ClearBoard concept by Hiroshi Ishii and colleagues.
We conducted a qualitative study of
MirrorBlender with three hybrid meeting sessions;
each with three co-located and two remote participants.
Participants were taken through a series of collaborative tasks,
like searching a map together.
These are two of our key results.
The first one:
Deictic gestures via one's mirror image
require mental effort, but are more expressive than cursors.
Using one's real body for pointing is not as simple as using a mouse cursor.
Due to the problem of parallax between the camera
and screen, users need to adjust the arm posture,
and pointing in this way requires mental effort. This is of course a limitation to this proposal.
However, the deictic gestures were very diverse and expressive.
With the ability to use both hands, participants came up with
interesting symbolic gestures. Like, when a co-located participant suggested a division
of labor in the map task by crossing the arms to indicate four individual map tiles.
The second result that I would like to highlight is:
Malleability supports inclusion of remote participants.
The ability to manipulate the position, size, and translucency of the mirrors enabled remote
participants to bring attention to themselves. And
this made them feel more included in the meeting.
For instance, remote participants often pointed in a bimanual gesture where one hand was used
for dragging their mirror in the interface and the other for pointing in the mirror image.
They further used gestures like wiggling their mirror to grab everyone's attention
or manipulating the prominence of their presence in the meeting
by changing the translucency and size of their mirrors.
Along with many more examples in the paper, these results lay out exciting new directions
for supporting deictic gestures and inclusive participation in hybrid video conferences.
Thank you for listening.