Hello!
This work is about Augmented Reality for television.
Specifically, it is about understanding what Augmented Reality is for home entertainment
in the living room.
Applied to television, Augmented Reality can be a lot of things.
For example, it can immerse viewers into an interactive storytelling space that enables
fantasy worlds to "break out" of the confines of the TV set as well as to "bring in"
aspects of the physical world.
As part of a televised show, the action can extend to the living room to create the sense
of "actually being there."
In this context, it is important to ask what ARTV is to understand the potential of AR
for television and to sustain growth in this area on solid foundations.
We make two contributions in this work.
First, we conduct a survey on Augmented Reality for television in order to understand past
efforts, critical ideas, and key projects.
We examine various perspectives on Augmented and Mixed Reality, from which we extract key
characteristics for ARTV and identify areas of investigation from TVX related to ARTV,
such as Ambient Media and Interactive Television.
We then introduce a conceptual framework called the "Augmented Reality Television Continuum,"
which is a two-dimensional representation space for concepts, devices, systems, and
applications for Augmented Reality Television inspired by the one-dimensional Reality-Virtuality
Continuum of Milgram and Kishino from 1994.
We also differentiate ARTV from Augmented Virtuality Television, a complementary concept
equally covered by our conceptual framework
Our method is composed of four steps.
In Step 1, we start from the perspectives and definitions of general-purpose Augmented
and Mixed Reality systems, examine current classifications, and overview research and
technical challenges.
This step enables us to provide an answer to the question What are the foundations for ARTV?
In Step 2, we perform a targeted literature survey of previous work implementing Augmented
Reality for television to understand What has already been achieved in ARTV?
Having established the foundations and understood the state-of-the-art, we proceed to identifying
relevant connections between ARTV and other areas of investigation from TVX, such as Multimedia
Alternate Realities, Ambient Media, or Interactive TV.
This step enables us to understand What makes ARTV a specific kind of television experience?
In Step 4, we identify key components for ARTV, which we build into our new conceptual
framework, the "Augmented Reality Television Continuum."
In our paper, we present a chronological examination of perspectives on Augmented and Mixed Reality,
from which we identify key concepts, implications for ARTV, and connected areas of scientific
investigation from TVX/IMX.
For example, in 2016, Ronald Azuma charcterized Augmented Reality as "a new form of media".
By adopting this perspective, Augmented Reality systems
are implementers of a specific kind of ambient media with characteristics from both the real
and the virtual world.
The implications for ARTV is that augmented television is a new medium for a new digital
media experience.
And the connected TVX area is Ambient Media.
Please see the paper for more details.
Our exploration has revealed many key characteristics of ARTV.
For instance, we saw that the Reality-Virtuality Continuum introduced by Milgram and Kishino
in 1994 represents the most accepted, go-to source for defining AR
and to distinguish Mixed Reality from Augmented Reality.
Rherefore, we adopt the Reality-Virtuality Continuum as the basis for our conceptualization
of Augmented Reality TV.
However, while the continuum of Milgram and Kishino can be successfully employed to characterize
the degree of mixture between real and virtual, its one-dimensional design is insufficient
for our purpose.
Instead, we need to consider the specifics of the television watching experience, where
a physical TV set is placed in a physical living room.
In our work, we recognize the fact that each of these two entities, world and TV, can be
independently augmented.
By adopting this perspective, we propose and introduce a two-dimensional conceptual framework
for ARTV.
The horizontal axis going from a completely real to a completely virtual environment is
actually Milgram and Kishino’s Reality-Virtuality Continuum that applies to the living room
environment (or the world).
The vertical axis, going from a completely physical TV device to fully virtual televised
content, is our adaptation of Milgram and Kishino’s Continuum for television.
Together, these two orthogonal axes characterize the various ways in which a television experience
can be augmented: in terms of the world, the televised content, or both.
In this conceptual framework, content can flexibly flow between the living room and
the TV set, while the living room and the TV set can independently flow across their
respective continua from real to virtual.
We call this space the "Augmented Reality Television Continuum."
To demonstrate the utility of our conceptual framework, we enumerate in the paper various
types of ARTV generated from the mixture of the two axes.
For example, the Conventional world and TV represents
the conventional TV viewing experience where neither the TV nor the room are augmented.
The corresponding point is the bottom-left of our conceptual space.
Another example is the Conventional world/conventional TV but with on-TV augmentation.
In this category, contextual augmentations are shown on the TV screen.
Examples include CollaboraTV, Audience Silhouettes, Teletext and Picture-in-Picture.
In this case, augmentation is perceived to
be off the TV screen, for instance on the wall behind it, in front or next to the TV set.
Implementations may involve hand-held devices, head-mounted displays, or video projection-based technology.
In the Conventional world/virtual TV category,
a physical TV is not needed, because content is presented either in a virtual TV frame
or even without a TV frame altogether.
Examples include TV projected on the wall, and when the storyteller appears to
be sitting in the living room in the viewers’ space.
Another case is where the living room is augmented directly.
This often requires spatial understanding of the surfaces present in the living room
for meaningful alignment of the computer-generated graphics.
Viewers experience an augmented world, but watch a conventional TV screen.
The world can be augmented and so can be the TV set, either by means of on-TV or off-TV augmentation.
The degree of augmentation of each component, world and TV, positions implementations of
this category at various locations in the ARTV Continuum.
Yet another category is achieved when the world is augmented, but the physical TV is virtual.
The Virtual world/conventional TV category resembles the Augmented Virtuality of the
RV Continuum of Milgram and Kishino: a virtual world is augmented by a physical TV set.
As a use-case, imagine watching your favorite TV show as a live video feed of your physical
TV screen, while wearing a VR headset but still being physically present in the same
room with friends watching the same physical TV screen.
Finally, the physical TV set is replaced by
one or more virtual screens.
This includes the scenario where a virtual TV is aligned to a virtual model of
a conventional TV set in order to recreate a familiar TV viewing experience in a virtual space.
Please go to our paper for more discussion and examples.
In our work, we found that ARTV can be many things and that prior work has implemented
it in various ways, from on-TV augmentations to offTV content visualized via AR-enabled
hand-held devices, HMDs, wall- and room-sized projections, and holograms.
By drawing from the various perspectives and key properties that we examined, we
can conclude that Augmented Reality TV reveals itself as a specific type of experience, immersion,
media, service, and gateway for televised content.
While the specific implementation may vary, what is important at this moment is to have
a rigorous basis to structure future research and developments, to be consistent in our
terminology, and to communicate our understanding of ARTV precisely by relating to proper frameworks.
Our ARTV Continuum is an attempt toward such a systematization.
There is more work that lies ahead, such as
understanding better the difference between Augmented and Mixed Reality TV, further exploration
of Augmented Virtuality Television, and connecting our framework to other projects, such as XRAccess
for people with disabilities.
We thank our sponsors for making this research possible.
Thank you!