Hi everyone. My name is Ornnalin Phaijit. I am also known as Lina. Today I will be
presenting our paper: A taxonomy of functional augmented reality for human-robot interaction.
Augmented reality (AR) technologies are today more frequently being introduced to Human-Robot
Interaction (HRI) to mediate the interaction between human and robot.
AR constitutes a vast field of research in Human-Computer Interaction, and
as it is picking up in HRI, it is timely to articulate the current
knowledge and information about the functionalities of AR in HRI
While some literature reviews have been published, so far no classifications have
been devised for the functionalities of AR in HRI.
Here we present an overview of HRI styles including how AR comes into play.
And, we propose a multidimensional taxonomy for AR in HRI that distinguishes the type of
perception augmentation, the functional role of AR, and the augmentation artifact type.
We place sample publications within the taxonomy to demonstrate
its utility. Lastly, we derive from the taxonomy some research gaps in current
AR-for-HRI research and provide suggestions for exploration beyond the current state-of-the-art.
As a bit of a background in regards to augmented reality,
we first must look into the reality-virtuality continuum.
The concept of a reality-virtuality continuum was proposed in 1995. Essentially, one end of
the spectrum is the virtual environment, which represents a completely synthetic environment.
The opposite end of the spectrum, represents a completely real environment.
Between these two ends is Mixed Reality (MR) in
which both virtual and real elements are present within the same display to the user. Within MR,
AR sits closer to the real environment’s end of the continuum
while augmented virtuality (AV) resides closer to virtual environment.
The difference between AR and AV is that the former
is referred to as the incorporation of virtual
objects into a real environment whereas AV is reserved for the
inclusion of real objects to a virtual environment.
Here is a diagram we built which provides an
overview of the current human-robot interaction styles.
We have kinesthetic interaction graphical user interfaced mediated interaction
simulation internet of things mediated interaction
teleoperation Now in the realm of
Virtuality, we have virtual reality or VR. The user resides in a virtual environment and
interacts with the robot which may reside in the real-world or virtual environment.
Similar to VR, but with an incorporation of real-world content, we have Mixed Reality.
Mixed Reality enables the fusion of digital and real-world content. On the left handside
of the spectrum of Mixed Reality is Augmented reality where digital content is superimposed
over the real-world environment to enhance the interaction between human and robot.
On the other hand, augmented virtuality is where real-world objects are instead
merged into the virtual world. A possible Augmented Virtuality scenario, for instance,
is one involving a real physical robot in a virtual environment.
In this work, we focus on AR in HRI, where we unfold
key interaction styles that can take place in that space.
The goal of augmented reality as a human-robot interface can be
to provide additional information to the human or the robot, or both.
The transfer of additional information can also be defined
as the augmentation of perception. We propose the classification of
perception augmentation types as the first dimension of our taxonomy.
the purpose of augmented reality can be divided into two categories:
1) to augment human perception, or 2) to augment the robot’s perception.
Note that augmentation of perception of both members may occur simultaneously.
AR can be used to modify the perception of the human. However, which part of the
human’s perception is augmented?
In the context of HRI, this category can be subdivided to accommodate the
three entities involved in a typical HRI task: the human,
the robot and the environment. An example of augmented human
perception of the robot can involve a robot wheelchair displaying obstacles in the
user’s desired path to clarify to the user why the planned path must be changed.
An example of Augmented human perception of the environment is this robot which
performs point cloud change detection in the environment and visually highlights
these changes to the user.
AR can also be used to augment the user's self-perception. For example, AR has been used to
visualise the user’s air-drawn model for a 3D robotic printing task
Just as AR can be used to augment the perception of the human, it can also be used to augment the
perception of the robot. An example of Augmented robot perception is when HRI
is enabled by integrating AR with a Brain-Computer Interface. To send
commands to the robot, the human gazes on the desired flickering option shown in the AR display
AR can also augment the robot's perception of the environment.
for example, it has been used to allow the user to define regions
of obstacles to the robot to supplement the robot’s perception of the environment
An example of augmented robot perception of self includes the ability for the user
to give feedback to the robot on its incorrect performance. This new
information from the human therefore allows the robot to update its comprehension
Although the classification in the previous section presents
the different places of perceptual augmentation, it includes
every minuscule interaction and hence does not describe
the overall high-level function for the user. We present this next dimension of the taxonomy
which fulfills this by highlighting the user-centred functional roles of AR in HRI.
Our first category is Artificial Timescale.
This functionality provides a virtual scene that is not based on the
present reality. It is used to alter the timeline - by using it to
imagine the future, have a glimpse into the past, or envision the impossibility.
An example is when the user is able to virtually fly the future version of the robot in real-time.
Next is Augmented Comprehension of the Present Reality.
In contrast to Artificial Timescale, in this functionality,
AR is used to augment the understanding of the present reality.
As an example, in a human-robot collaborative cardoor assembly task, AR has been used to
visualise to the human the objects hidden inside a toolbox.
Our last category is Augmented Control.
In this category, AR is used to provide robot control to the user.
For instance, users were able to control the
robot by using tongue movements to select options in an AR menu.
This dimension has now given us the "why" for AR in HRI.
Now we present the next dimension which will examine the "how".
In AR, virtual objects are presented to the user on the user’s display to deliver information.
These artifacts can be classified based on their functions:
Augmented Embodiment, in which the artifact is placed:
on the robot body, on the
human body, or as a whole new robot.
Next is Augmented Interactive Objects.
These are virtual objects placed in the real world
which the human or robot could interact with.
Augmented User Interface refers to cases in which the
virtual objects are fixed in place on the user display
regardless of the orientation or location of the user
viewpoint.
Lastly, Augmented Scene refers to the remaining virtual objects
which do not fit into any of the aforementioned categories. They
simply contribute to present the scene on the AR display.
Using our taxonomy, We categorised the 20 most cited publications
in AR-for-HRI between the years of 2011-2021.
As shown in Table 1, all studies but one fall under all three subcategories
of augmented human perception. This suggests that high importance
is placed on the augmentation of human perception. This is as expected as AR technologies
are primarily used to aid the human experience.
The range of human perception augmentations therefore tends
to be more well-researched than that of the robot.
All but three studies utilised the Augmented Control functionality,
suggesting that AR is a recognised mediator for robot
control.
Although the number of papers examined here is not exhaustive,
the table suggests that there could currently be a gap
in the most cited research for the robot’s side of perception in
AR-for-HRI.
From our work, we also found some potential research directions for
AR specific to the HRI context:
- Although augmented human perception is widely employed in AR-for-HRI,
it may be insightful to examine how augmented perception is used.
With ongoing research in human-in-the-loop learning agents,
bringing AR into the field could be advantageous as it expands interaction capabilities.
For future research,
it may also be insightful to investigate if there is a link
between the functional role of AR and the robot’s task,
or the relationship between the artifact type and the task of the robot.
To take the full MR spectrum into acount, Further work will need
to be undertaken to include AV. Currently, AV is largely unexplored in HRI.
With our proposed categories, we hope to help future research
find inspiration to explore the various functional scenarios. We also hope that our proposed taxonomy
can be used to help future research describe the functional roles of AR as well as the artifacts
used to achieve these functions. Thank you