Hi, I am John Joon Young Chung,
from the University of Michigan.
Today, I am going to present you the project that investigates
an approach to allow scalable visual communication
from viewers to the streamers in a live streaming setting.
This work is done as my internship project at Adobe,
with great collaborators,
including Valentina, Haijun, Li-yi, and my mentor, Rubaiat.
So, live streaming is on the rise,
with expansions in the number of users and platforms.
The domain is also expanding,
from gaming to creative live streaming and knowledge sharing.
One of the key components of the success of live streaming
has been interactions from the viewers.
Traditionally, user chat messages have been the main way of
viewer communication.
However, as live streaming is visual in its medium,
viewers communicating through visual means
have been considered as an opportunity.
Existing commercial systems and previous research systems
have shown the value of it,
showing that visual viewer inputs can allow
viewers to have richer communication.
However, there is a challenge in using viewer visual inputs
in scalable settings,
as many viewer inputs can easily overload the information channel.
While there have been some approaches to handle the scalability issue,
but they were either limited to simple visual inputs,
like clicks on the video,
Or only worked for a limited set of live streaming settings.
So, our core question was
how we can support more flexible scalable viewer visual communication
in live streaming.
To understand the problem better,
we conducted formative interview
and identify four design goals.
First, allow viewers to have more flexibility
in expressing visual inputs.
Second, collect relevant viewer inputs
and present them in an easily understandable way.
Third, visual inputs should be easy to make,
so that viewers can participate without hurdles.
Fourth, allow streamers to adapt to diverse live streaming settings.
For more details about the formative study, please refer to the paper.
Based on these design goals,
we introduce a framework
to design scalable viewer visual communication systems,
Visual Input Management Framework, of VIMF.
VIMF considers each viewer visual inputs
as a combination of visual attributes,
which can be used to convey information.
For example, position, shape, rotation, size, or color of visual input
can be used to convey certain information.
With these visual attributes,
VIMF first allows streamers
to specify these visual attributes in viewer inputs,
Let us explain this with an example case.
If a streamer wants viewers
to annotate the force of friction applied on the box,
the streamer might want viewers to only draw an arrow.
So, the streamer can set the shape of visual input as the arrow.
However, for the position, rotation, or length information,
the streamer might want users
to freely place arrows with these aspects.
Then, for these other aspects,
the streamer can impose no restrictions.
The system with VIMF then collects visual inputs
from viewers with these restrictions.
The second feature of VIMF is to aggregate viewer inputs
according to visual attributes that are critical.
In the example, the streamer might think
the most critical information is
the rotation, or direction of the force,
and the length of the arrow, the magnitude of the force.
In this case, the streamer can only choose
these visual attributes
and cluster viewer inputs only considering those aspects.
Third, the streamer can visualize these aggregated viewer inputs
so that both the streamer and viewers
can better understand collected inputs.
The streamer can combine this visualization
closely to the underlying video,
so that it can be effectively conveyed.
In the example, aggregated viewer arrows can be placed on the box,
more clearly explaining
how viewers actually have submitted their visual inputs.
So, this is the basic framework,
and based on this, we designed a system VisPoll.
VisPoll focused on specific visual attributes,
that are about single, static objects.
Hence, it only focused on visual attributes of
shape, color, size, rotation, and position.
It was also designed to have “turns”
when users are allowed to make visual inputs,
so that streamers can more easily manage viewer inputs.
Moreover, streamers could save their configurations
of visual attributes, aggregation, or visualization.
They can also use templates
so that they don’t have to do configuration interactions
while they are doing the streaming.
For more details on how specific interactions can be done in VisPoll,
please refer to the paper and the accompanying video.
We demonstrate that our system can be used
in various application scenarios,
from education to creative live streaming.
With VisPoll, we conducted a preliminary evaluation
with three streamers and 43 viewers.
We could find that streamers
used VisPoll according to their live streaming context,
while allowing easy and expressive visual interactions with viewers.
Moreover, streamers found values
in aggregating and visualizing viewer inputs,
as they can help understand the overall viewer inputs,
while further engaging viewers into the streaming.
In this paper, we introduced a new opportunity
to use visual communications with expressiveness and scalability
in the live streaming.
We hope that this work can inspire HCI researchers
to further explore multi-modal communication channels.