Hi
The idea of applying AR in TV broadcasting
has been around for at least the past couple of decades.
Researchers have been testing its application in all three stages of production, distribution, and consumption.
On the consumption side in particular,
application of AR is promising.
This often consists of using AR to deliver
virtual artifacts or extra information in the viewers environment,
potentially enabling the viewer to interact with these artefacts.
This can enhance or transform the conventional TV viewing experience.
Commercialization of AR devices suggests that this application may gain more traction in the future.
However, it is not clear how to design content for this paradigm.
What are the parameters that the designers need to consider,
when creating content for this type of presentation.
To answer this we conduct a systematic literature review and look at the peer reviewed publications in the field.
We look at SCOPUS, ACM DL, and IEEE.
We select 42 papers for analysis that describe systems, prototypes or scenarios that match our criteria.
For more information on our methodology please refer to the paper.
We capture often implicit design decisions in a set of dimensions.
These dimensions consist of a set of options that capture the relationships between various elements
within content,
between content and the world outside: i.e. context,
and between various elements within context.
Context related dimensions will be covered in forthcoming publication,
the rest are explored in more detail in the paper.
Today I will talk about three content level dimensions:
abstraction, time, and display;
and a content-context level dimension: interaction.
From a technology PoV, content can be classed in two groups: AR content, and TV content.
Here, AR content is all media artefacts that are presented using AR technology,
TV content is the rest of the media artefacts that already existed prior to adding AR.
With this in mind, now we can talk about Abstraction.
Abstraction captures the relationship between AR content and TV content
regarding their contribution to make the overall content meaningful.
When thinking about AR content and TV content individually,
if they are meaningful and complete independently, then the abstraction is independent
If the completeness of the experience requires both AR and TV components,
then abstraction is dependent.
If TV content is complete and meaningful on its own
but AR content is used to deliver extra features or to enhance the experience,
then abstraction is additional AR.
The reverse of this, where AR content is meaningful and complete on its own
and TV content merely enhances it, is additional TV.
We found that most works fit into dependent or additional AR classes.
For instance, Revelle et al. created a system where children would watch TV content for a few minutes,
then the TV would be paused and the children would be asked to interact with the AR component of the show.
Once they captured relevant AR artefacts and threw them back into the TV screen,
then the TV content would get resumed.
You can see that both AR and TV components are required to make the overall experience complete.
In case of additional AR, for instance, vatavu created a system where silhouettes of remote
viewers were superimposed on TV content.
Here, AR content is adding value, however, without them, the TV content remains complete and meaningful.
Another content level dimension is Time.
It captures the relationship between AR content and TV content regarding their timelines.
While asynchronous presentation of AR and TV content may be suitable for experiences
that present independent AR and TV components,
in most cases, typically it is desirable to have synchrony between the two components.
Naturally researchers have tried to invent ways to synchronize AR content with TV content.
This has resulted in two ways of delivering content in a synchronous way:
Continuous and intermittent
Continuous delivery requires both AR content and TV content to be presented simultaneously to the viewer.
For instance, Vatavu’s Audience Silhouettes
present AR artefacts while the TV content is being presented.
On the other hand, Revelle et al.'s system
required the viewers to go back and forth between TV content and AR content, intermittently.
The final content level dimension that we will cover today is display.
When using AR in the consumption phase of TV broadcast, we found three ways in which
content can be displayed: either both AR and TV content are presented on the same display,
or they are presented on separate displays, or the TV is eliminated and an AR display
is used to display content.
In case of the same display, for instance, Vatavu’s Audience silhouettes are superimposed
on top of the TV content and displayed using the same device.
In case of separate displays, an AR device
for instance, a hand-held or a head-mounted display can be used to present AR artefacts.
For instance, Baillard et al. used both of these to present AR artefacts in the area around the TV screen.
TV via AR removes the TV set entirely.
There are instances where an AR display is used to present TV content to the viewer,
in one or more virtual frames.
This can be used, for instance, to anchor TV content to a physical location.
Another option here, is to present content outside any frame.
For instance, using chroma-keying or volumetric capturing techniques,
TV content can be presented in an un-framed way and appear to be present in the viewer’s environment.
Moving on to a content-context level dimension, we can talk about interaction.
Interaction captures the relationship between the viewer and content. It can be seen as
a set of behaviours that enable the viewer to dynamically change an aspect of content.
We found that there are three levels of interaction in the works:
display level, structure level, and content level.
Display level interaction enables the viewer to change the programme, resize, reposition,
or even change camera angle.
For instance, Baillard et al. created a system where thumbnail images of other programmes
were displayed next to the TV screen. The viewer could drag a thumbnail and drop it
on the TV screen, to change the programme.
Structure level interaction enables the viewer to influence the story path and break away
from the linear structure.
There are not many examples in our review that aim for this type of interaction,
but given the success of programmes such as Bandersnatch, it is likely that this level of interaction
gains more popularity in the future.
Content level interaction is akin to how players interact with game content. It requires direct
manipulation of artefacts to achieve a goal.
For instance, in Revelle et al.’s system
the children were tasked to capture words in their written and visual representations
and throw them back into the TV screen, to resume the TV show.
OK so we talked about three content level dimensions: abstraction, time, and display
And we talked about one content-context level dimension: Interaction
Please refer to the paper to see a more in depth explanation with more examples of these
and other dimensions.
The dimensions and the options they encapsulate, can be combined to create unique patterns.
For instance, here I’ve selected one aspect of each dimension to create a pattern.
This pattern provides the parameters of a concept that mixes AR with TV.
We can now write a scenario for an ARTV programme.
For instance, imagine a crime TV series; Between each episode that is broadcast on TV,
the viewers can take part in the investigation and search for evidence and clues.
The viewers may be asked to go to a specific location in town, where a witness claims:
she has seen the suspect on the night of the incident.
They can use their AR display (e.g. smartphone)
to find and scrutinise the evidence. Once they submit theses artefacts, the criminal
investigation department in the story-world can take over and examine the evidence in
the next episode, thus carrying forward the story.
In summary, we talked about using AR in TV broadcast,
We talked about dimensions, we explored a few dimensions in detail.
We presented them as a way to create unique patterns, and we used a pattern to create
an ARTV concept.
This can be a method to explore the design space and to generate guidelines.
This framework can help and structure the conceptualisation and ideation of ARTV concepts.
Thus enabling broadcasters and designers to ask the right questions when creating such
programmes.
In the end I would like to thank my co authors and the institutions that provided the support
for this research to take place.
Please contact me for any questions or drop me a line if you’d like to talk or would
like to collaborate on topics such as media, TV, or any form of R.
Thanks I’m Pejman Saeghe, and this has been
Augmented Reality and Television: Dimensions and Themes.