Hello everyone, my name is Guillermo Reyes, and I’m here to talk about multi-modal multi-scale
attention guidance for cyber physical environments or CPEs for short.
In these environments, users often need to
find specific places and objects that may or may not be in their immediate vicinity.
The larger and more complex they are, the more difficult this will be as users become
distracted by their surroundings. This is why guiding the user’s attention
is so important. Attention guidance is the deliberate shifting
of a user’s focus of attention towards an object or away from distracting ones.
How can an environment do this?
We could shift a user’s focus of attention
by making an object in the environment salient. Without dedicated devices this could be difficult,
but what we could do is make use of the visual and auditive devices that are already in these
environments. Unfortunately, some devices will be effective at a certain distance or
location, but not at another. This is where multi-scale attention guidance comes into
play. By taking into account different parameters about the user and the target as well as the
different devices’ properties, we can make a decision on what device is the most suitable
to guide the user towards a target object. Our work proposes an attention guidance method
that ranks the suitability of visual and auditive devices to guide a user’s attention towards
a target.
Now, what is a suitable device?
We could imagine how suitable a device is as a function of the user the target and a
set of context parameters. But even if this function existed, we still don’t know what
it looks like. Whatever it may be, one could argue such a
function would at least consider 2 things: how salient the device is and how close it
is to the target. While closeness asks question “How easily
can a user find the target using this cue?” salience asks “Does this cue attract the
user attention?”. We call these two primary functions and, in turn, each of them have
secondary functions. I will now talk about what is meant by these
two terms but I’ll focus more on the visual modality.
Let’s start with closeness.
We tend to think about how close two objects are in terms of distance in km, m, cm etc.
But when guiding a user’s attention towards a target it is also important to consider
how easy it is to find the target inside area highlighted by the cue. We assume that the
more similar the highlighted area and the target are in size, the easier it will be
to find the target inside the area.
This describes what is meant by both terms: on the top left is what the perfect cue would
look like in terms of closeness. To its right is perhaps the next best kind of cue when
we are far away. If we are close enough, perhaps bottom left would be good if there is no better
cue but that depends on how far off it is and it should be avoided if possible. bottom
right might better be avoided as it tells the user to look for the target where it is
not. The user will just lose time. The function could look like the one from
suitability before, where we also have a weight to tune how important each of these terms
is for our concrete scenario.
Now, how can we calculate accuracy? we were
to consider accuracy in absolute terms of distance, cues that are near the target would
always be the most accurate while those farther away would barely have any accuracy. This
could lead to problems if the user does not see presented cues if they are too far away
or to cues guiding the user in the wrong direction. To counter this problem, we dynamically adapt
what is understood as accurate by introducing a sigmoid as a function of the distance from
the user to the target. That way when the user is far away, cues that are nearer to
the target are still more accurate, but cues that are farther away are still considered
relatively accurate. As the user moves closer to the target, what is considered accurate
changes decreasing the accuracy of cues that are farther away.
For precision, the story would be similar.
When we are far away, we are less concerned about the precise location of the target.
A rough area of where it is suffices. But as we get closer to the target, it becomes
more and more important that the area where the user has to search in becomes smaller
and smaller.
Also, Just like there are secondary functions
for closeness, we also have secondary functions for saliency. Saliency is based on contrast.
Salient objects are ones that are different from their surroundings. Things that just
pop out. While there are many features like intensity, color, orientation, etc., which
can be salient, here we focus only on two: brightness and size. Furthermore, we make
some assumptions about the environment and what is salient in them. In the case of visual
cues, the first assumption is that brighter cues attract more attention. The second one
is that larger cues attract more attention. We then just scale the brightness and size
of the cue to a range of 0 to 1.
The auditive modality is very similar to the
visual one. Regarding closeness accuracy could be considered the same. But we could make
a difference in precision. Precision referred to the search area where the user looks for
the target. For the auditive modality it could be interpreted as how good does the cue point
us in the right direction. We could apply a sigmoid function in the direction from the
user to the target within certain bounds. If the cue points the user in exactly the
direction of the target, then this would have a precision of 1.
Regarding salience, we could make the assumption that louder devices attract more attention.
One could, thus, determine how salient an auditive device is in a similar way to brightness.
By calculating the sound pressure level we could predefine a certain number of decibels
that make a cue 100% salient.
All I have talked about so far is how to get
the most suitable device among a set of devices for a user and a target. What I have not yet
talked about is how this device changes over time. If we always selected the most suitable
device, it could be possible that devices just start flickering all over the place confusing
the user. This is because the environment could be split into hard regions where each
region is dominated by one device. To avoid over-switching guiding cues, we introduced
a margin. That way, we only switch the guiding cue if we find a device that more suitable
than the one that is currently selected by this margin. Additionally, since we eventually
want to use the most suitable device, we reduce this margin over time.
Finally, the current definition implies that
devices could be either only close or only salient. Cases where some cue is too salient
but not close to the target would guide the user away from the target. Cases where the
cue is too close to the target but is not salient will not attract the user’s attention.
What we want is something that is both. That way, a sequence of suitable cues would eventually
lead us to the target. By adjusting the interaction weight, we can decide how important this is
in any particular CPE. If it’s 0, then the definition is the same as before. If it’s
1 then only the interaction term matters. If it’s in-between then the individual terms
can still make a difference.
To test our system, we conducted a user study
under different conditions. We had the hypotheses that the mixed modality would provide better
performance in terms of distance traveled. We also thought that the lack of an interaction
term and a lack of margin would both lead to worse guiding performance, meaning higher
distance traveled. And we also thought that the lack of a margin would make the system
more volatile in terms of a larger amount of cues given to the participant.
We divided the participants into two groups for the mixed and visual-only modalities and
they then completed a series of trials searching for objects under different interaction weights
and margins. A margin of 0.2 was selected, meaning that a cue had to be at least 20%
more suitable than its predecessor to be selected.
We evaluated this in the context of a VR supermarket. The participants had to move around the supermarket
looking for the target products while they were guided by different types of cues. These
would be a ceiling lamp, an electronic label, or series of electronic labels or a speaker
placed behind each of the virtual products that would beep to attract attention.
What we got were mixed results:
When margin is fixed at 0.2 a better performance is found in the mixed modality independently
of interaction weight. But this was not the case when there was no margin. Furthermore,
when a margin of 0.2 was selected, performance in the mixed modality was slightly improved,
while in the visual-only modality it got worse. We don’t yet know why this was the case.
There were a significantly higher number of cues in the no margin condition, which confirmed
our hypothesis of the no margin condition being more volatile.
The interaction weight did not have a significant effect. So either the parameter did not have
a strong influence in guiding the participant or the design of the supermarket did not present
enough opportunities where this could be used. We should remember this is a VR environment,
and most of the participants had never had such an experience before.
To conclude, we introduced a novel method
to guide attention in cyber-physical environments which tries to assess the suitability of a
device to guide attention. This method consists of primary and secondary functions. The advantage
of this is that the secondary functions can be replaced for better ones without changing
the overall method. The initial study we conducted showed some very promising results in the
effectiveness of this method, but also some artifacts than need further testing. This
of course opens up many new research opportunities like testing the system in a real environment,
coming up with new functions or investigating how this method could be applied to a multi-user
setting.