Hello, and welcome to this presentation detailing our work
entitled towards future pedestrian vehicle interactions
introducing theoretically supported AI prototypes. I am
Wilbert Tabone from the cognitive robotics department of
the Delft University of Technology. And I have worked on
this research for the SHAPE-IT project together with my
department colleagues at TU Delft, Joost de Winter and
Riender Happe, and colleagues from the Institute of transport
Studies at the University of Leeds, Natasha Merat and Yee Mun
Lee. In the future urban environment, automated vehicles
will be communicating with a diverse set of interaction
partners, and an interaction loop which enables a collective
behaviour that combines safety and efficiency for the other
partner of this interaction loop, which is the human. A
particular subset of other road users are vulnerable road users,
which include pedestrians and cyclists amongst others. Now, in
order for the interaction loop to be complete, the AV must
communicate its intentions to the vulnerable road user, which
traditionally was communicated explicitly through the driver or
also implicitly through vehicle kienamtics . Now, whilst the
latter would be retained explicit communication would be
lost without a human driver at the wheel. One such solution to
this problem are what are known as externa; human machine
interfaces (eHMIs), which are attachments to the exterior of
the vehicle that communicate with different modalities.There
have however, been some drawbacks for eHMIs, such as the
ambiguous situation created when multiple vulnerable road users
are encountered by an AV or even multiple AVs are present. And it
might not be that clear to whom the messages being communicated
to. Moreover, since multiple eHMI designs might be uncounted,
perhaps even in different languages or these are located
on different parts of the vehicle, there is the risk that
there is no understanding or it's not understood or located
and seen in time. All of this also boils down to the issue
that there is no standardised design for eHMIs. So a potential
solution to this problem could be a shift of the interface by
using augmented reality technology, which has already
been proven useful in several domains. As the AI becomes more
context aware and pervasive, perhaps with the consumer onset
of AI vehicles, its potential to benefit pedestrians will become
clear. Through augmented reality, the interface could be
removed from the vehicle and placed anywhere in the world,
which has the potential to alleviate or solve the problems
faced by eHMIs. Each user would get direct feedback and then the
users own language. other advantages that this solution
presents are cheaper costs and an easier route to
standardisation. There have been several examples of previous
application of AR in traffic both for the driver side and for
the vulnerable road user aside. However, there are in fact very
few augmented reality prototypes for pedestrian AV interaction in
a real environment, and therefore it is also currently
unknown How should such prototypes function and look?
This paper attempts to address this research question by
presenting augmented reality prototypes for pedestrian AV
interactions that function and alien environment. These
concepts were device derived by building upon interviews with
human factors experts presented in our position paper published
earlier this year, and the prototypes were designed using a
theoretically informed approach that draws upon fundamental
principles and concepts such as affordances, the field of safe
travel and risk perception. The following is an overall process
followed consisting of four phases. First, the perspectives
paper and its corresponding interview transcripts were
analysed and seven quotes which could be turned into
implementable concepts were extracted. This was followed by
a process where literature which could serve as a source of
inspiration, or is a theoretical foundation for the design
process was collected. before deciding on the concept, the
scenario was chosen to be an unmarked two way road and more
than two states vehicle is not yielding and vehicle is yielding
were chosen, together with red and green colours to communicate
these states as studies show that these are the most
intuitive for pedestrians judging the intentions of an
approaching vehicle.
In the third phase, brainstorming design sessions
were conducted amongst the authors using the MIRO
collaborative environment where for each concept, design
elements were discussed at length, with reference to the
literature identified for the concept and decisions were
placed on the affinity board on clustered post-its. At the end
of this process, nine designs emerged from the seven concepts
from the seven courts of fidelity paper sketch was
created for each and the specific AR heuristic evaluation
was conducted to identify any shortcomings before
implementation. Implementation was then carried out using the
Unity MARS environment and developed for the iPad Pro due
to it's inbuilt LiDAR. Unity MARS enabled for testing the
concepts both in an environment create edited, created in the
editor and in the real word site and allowed for AR apps that are
context aware and responsive to physical space using various
fuzzy features. The final iPad application uses the camera and
LiDAR to scan the real environment generate planes of
the world and sport the concepts of the fuzzy conditions set such
as the horizontal surface of a certain size I met and here they
are denying AR concepts. For the first concept, we decided to
change the surface and colour of the road. Similar non-oriented
concepts have used LED embedded in the road to display crossings
on demand. And an in-vehicle interface the showed a red or
green carpet on the motorway depending on whether it was safe
or unsafe to change lanes. The other concept served as an
inspiration to use green bars for the yielding state. We chose
a zebra crossing design rather than colour on the road so that
there is no instructive suggestion to cross. Here we
present a concept that changes the vehicle's surface through
the addition of a plane that appears in the windshield area,
since this is where people mostly look when the vehicle is
approaching. We also added redundancy gain by superimposing
icons to avoid any confusion, where a red plane may be
perceived as yielding rather than green. Concept 3 expands on
the previous concept by drawing upon the theories of looming and
conspicuity. Looming is the human response to the rapid
approach over sort of the body, while conspicuity is defined to
be something which is obvious to the eyes striking and attracts
attention. The coloured planes in concept 2 were modified and
as you can see, the non yileding state approach grows as it
approaches the user and vice versa for the green state.
Concept 4's idea was to create a safe corridor for the pedestrian
and hence we drew upon the field of safe travel that was proposed
by Gibson back in 1938. This field at any given moment
represents the path that a vehicle may take unimpeded and
where it would be dangerous for a person to walk. The concept
also relates to the theory of affordances since the shape of
the field reflects physical laws. Here we see the concept
implemented in the real environment. Concept 5 does not
need much explanation as it is just an augmened binary
pedestrian crossing lights, which is attached to the world.
For concept 6, we needed to augment a 3D traffic light in
the form of a virtual fence. This concept became more of a
tunnel analogous to the tunnels in the sky concept in
conjunction with the safe corridor augmented in-vehicle
HMI which demarcates which area of the road should be avoided.
This is also similar to the carpet metaphor explored for
concept one. As can be seen, the tunnel is red when the vehicle
will not yield and a red gate blocks the pedestrian's path one
on the other hand, the yielding state is triggered when the
gate which switches to green and is opened to allow for the
pedestrian to cross. Concept 7 was inspired by the previously
published vehcile intents concept which highlights the
vehicle's path within an arrow and stopping line. We built upon
this concept by including an augmented representation of a
hatchback car similar to what can be seen in the ghost replay
system in a racing digital games or Tesla's shadow mode. We call
this the Phantom car as it is basically an augmentation of the
car in front of the vehicle, which indicates where the
vehicle will be after a particular time in the future.
For concept 8 we developed a HUD textual interface to alert the
pedestrian. The difference from the previous concept is that the
interface moves with the pedestrians field of view, the
message is shown using non-instructive language and
icons are added for redundancy again. And the final concept in
a way combines both concepts 5 with 8 and the pedestrian
traffic light has now been detached from the world and it's
floating around and the interface element follows the
user's field of view. As previously discussed, for each
concept, there were no direct instructions communicated to
avoid erroneous crossing decisions and concepts not
augmented on the ground were made of translucent material not
to block the user's view of the real world beyond.
Moreover, each concept did not use colour alone and supporting
cues such as movement or iconography were added to
facilitate redundnacy gain. Now onto the limitations. It is to
be noted that several assumptions were made and hence
the presented concepts could not be viewed as fully furnished AR
interfaces thatbulnerable road users can already use. The
implemented concepts and videos take only one vehicle into
account. Moreover, while augmentation was intelligently
handled through MARS, the state change for vehicle intent was
manually triggered using a tap on the iPad. Due to the nascent
status of this technology, various workarounds had to be
implemented for concepts that required knowledge of the
position of the approaching vehicle and the speed. These
include having thr real vehicle remaining static as moving real
world objects are not yet supported by the libraries that
were used. While the road position was handed by Unity
MARS through the camera and LiDAR, vehicle position was
hardcoded, or image markers were used and relied upon to spawn a
concept. In the future, such drawbacks could be solved
through edge computer vision methods, which could be used to
detect the distance and speeds of approaching road users.
Moreover, user studies are needed to identify which
concepts are best accepted by users, how the concept would
function on augmented reality glasses in an urban environment.
We also identify the state's switching figures and the
intuitiveness of the mappings used and it would also be
worthwhile to explore how to integrate AR concepts with other
communication methodologies for VRUs such as eHMIs and also also
to create provisions for VRUs with visual impairments.
Finally, although we used an iPad Pro, the concepts could be
easily exported to a head mounted display. In such a case,
the following sensor requirements posed by each
concept would need to be met. In conclusion, our paper presents
an outlook on future AV-VRU communication through augmented
reality in the form of prototypes, and explores how
these could function and look like. A four phase process was
employed to create nine prototypes of such concepts
which function in the real environment. In the spirit of
open science, we have made the code source available to
facilitate reproducibility and to encourage further
development. Thank you for your attention, that I now look
forward to your questions, but please feel free to get in touch
by email or on Twitter if you have any further queries. Thank
you very much.