- Good evening, this is my paper on peripheral vision
basically for smart glasses with head-mounted display.
I wrote the paper at the end of my undergrad
in collaboration with HKUST as well as Deutsche Telekom.
My professor belongs to both Helsinki as well as HKUST.
As you know, smart glasses are becoming really popular.
They're used for information, visualization,
medical, as well as navigation purposes or gaming.
But one of the biggest problems
is the limited field of view,
so if you have seen the smart glasses
like Google Glass, they'd want a display so small
that you have to really look into the screen
and stare all the time.
So we introduced a mobile peripheral vision model,
which can be used on smart glasses
with head-mounted displays like Google Glass.
And I'd send a software solution,
and basically we simplify multitasking using this model.
So we used this model to build our navigation application
for smart glasses for both indoor and outdoor scenarios.
So before going into,
I just want to introduce some terminologies real quick.
Angular field of view is basically measures
your extent of a 360-degree circle
that is visible by your human eye.
And your peripheral vision, it's basically outside
your central system and why is it important
is basically your peripheral vision is responsible
for your night vision,
as well as for your motion perception.
The angular field of view of Google Glass
is pretty small, it's like 30 degrees,
and this limited field of view, it forces the users
to really focus his eye gaze towards a small screen
to extract the meaningful information,
and this can really create a lot of visual fatigue.
Moreover, I'm not sure how many of you have seen
the Google Maps interface in the Google Glass,
but it's really small, and even if you're just walking
or even using by biking,
it's really difficult to use this app.
And basically, Google Glass screen,
it occupies between the near-periphery
and the mid-peripheral area.
So first, it blocks your peripheral vision,
and second, if you're really putting
a central gaze into the screen,
then you also block your central vision.
And this can create a lot of fragilities
because you need your peripheral vision
to know where do you have to put your step down.
You need your peripheral vision to see
where the obstacle's located.
Similarly for the mobile devices,
it really limits a lot of cognitive ability,
and restricts your peripheral vision,
especially if you're walking.
There have a lot of pedestrian fatalities because of this.
So there's been a lot of work done to try enhancing
the field of view of smart glasses,
especially most of them have used, like,
LEDs on both sides of your glasses,
and most of the solutions have been very hardware-oriented.
Some of them have tried to talk about
how you can use motion perception at the periphery area,
but they have only talked about how you can use
on static users, they're having a lot of work done
on how you can redirect your central gaze
towards the glass screen, but there have been no work
on how you can actually do multitasking.
There's one paper which was publishes in 2016 in CHI
which is very similar to our work,
and our work is actually built up on top of it.
The work actually talks about the five guidelines
that you need to basically present information
at the periphery.
It talks about how you should use simple shapes,
colors, instead of using some composite shapes,
and how shape recognition is independent
of visual motion perception at the periphery.
I'll explain in a bit what that means.
But the solution that they provide
is basically they put two smart glasses
on both sides of your smart glass and it's bulky,
it's more of an hardware solution,
and it's more for static users,
whereas we try for different lighting and backgrounds
for both indoors and outdoor.
And it's a software solution you can deploy
in pretty much any smart glasses.
And we assume that the smart glasses
that we are considering are located
between your-near peripheral area,
but you can always customize it,
and I explain it why soon.
So the contribution of this paper is we provide a model
which uses color and motion to display the visual cues
in the peripheral area.
And then we develop a high-fidelity peripheral vision
navigation application for both indoors
and outdoor environment scenarios.
And we discuss two cases, strabismus and color-blindness,
for which our model does not really apply.
I explained this already, we use limited graphical language
and we focus a lot on multitasking.
So the theories, we looked into two basic theories,
firstly the color detection,
and the motion detection in the peripheral area.
So just going and digging into it a bit,
so there are two types of cell in the human eye,
the cone cells are really responsible
for color red, green and blue, and the rod cells
are really important for your motion
and night detection at the periphery,
that's why it's really important.
So most of the red and the green cells
are located in the center.
Red occupies around 60 to 70% and green around 20%.
And the blue cells are mostly at the periphery,
so that's where your blue perception comes from.
And so we have tried to use these three simple colors
to signal information to the users.
Blue because it's the highest number
of cones in the periphery.
Red because it's the highest number
of cones in the retinal center,
as well as green because its high contrast
with other two cones.
And we tried to use motion detection
by the peripheral vision using simple shapes.
So we digged into it a lot and we figured out
that the rate threshold of velocity
for visual motion perception,
it increases as we go far from the central gaze.
So if an object is located far from your eye,
you need to throw it at a much higher speed.
So basically for our case,
what we've basically assumed that most of the glass screens
are located between near-periphery and mid-periphery.
But if some glass screen is located further far,
then you can adjust the velocity accordingly.
We build up a very simple app initially using this model
because we wanted to see how you can use this model
for indoors for places like museums,
for city park, because as a tourist,
if I wanna look into more attractions
rather than just keep focusing on the screen,
either mobile screen or glass screen,
it's really not useful.
So we made up a very simple app
where if the blue color blinks on the left side
to indicate a point of interest,
then you basically turn left.
And once the user turns left, the entire screen turns blue.
We conducted a simple experiment to see how it works.
And then we kind of saw that
there are other applications to it,
like video games, outdoor AR games, and notifications.
For video games, you can show something like
a red color blinking on your right,
and you can show the enemy in that way.
For notifications, it really depend how complex
are your notifications, but I'm sure you can adapt
your app according to that.
So based on that, we build up a demo applications,
we went through a lot of iterations.
What we did is for indoor application,
we used indoor Wi-Fi localization method
in that sense that we basically used
the Wi-Fi access points and the signal strength
and we use the best five Wi-Fi access points.
And to detect the particular location,
we had a customized part which was
around five to 10 minutes long and we built up
a HashMap using these Wi-Fi access points.
And we made sure that all the signal strength
that we are using for the Wi-Fi access points
are more than 90 decibels, minus 90 decibels,
so it means the signal strength should be good.
And for outdoor, we use GPS
which has a precision of, at most, five meters.
For indoors, we tested this app for both,
we deployed the app on mobile,
as well as on the smart glasses.
For outdoor we were using MAD Gaze,
which is used a lot in Hong Kong.
And it basically has a quad core process serve,
it's pretty good for multithreading activities
and outdoor scenarios, so we used that.
So this is the simple, how it looked like if it's blue,
which means the user continues to walk straight,
if it's red, then the user turns right,
if it's green the user turns left.
Now as I mentioned, we are just not using
only the color perception,
we are using two different types of information,
first the color and the motion at the periphery
to display information to the user.
Before going into the experiment,
so we did a lot of iterations to it.
With every iterations,
we actually picked 20 different users.
We actually had a balanced data set
of 50% males and 50% females,
and most of our users were first-time users
of any smart glasses.
70% of the users had eye power,
which means they were using glasses
on the top of their own glass.
And, I guess, oh,
so yeah.
So what we did in this case,
you'll see the navigation hint bars
which will stimulate your peripheral vision through motion.
The bar movement basically covers the entire 30 degrees
of your field of view Google Glass,
and these hint bars, they basically blinks four times
for two seconds, which means your bar blinks
at the speed of 15 degrees per second.
This 15 degree per second is well above
the human threshold, which is 2.15 degree per second,
at 90 degree eccentricity.
Eccentricity, what I mean by eccentricity
is how far your object is located from your central vision.
So basically, it's gonna blink like this.
And then, as I told, because it blinks four times
for two seconds so the total peripheral stimulation
is two seconds, and this is well above
the reaction time that a human needs to rapid movement,
which is at least 250 milliseconds.
It's the same thing with turn for the left,
and the cue is activated with a radius
of three meters from each turn.
Now, initially we were only doing for indoors,
but we went ahead to do with outdoors
because we figured out the Google Maps
which is deployed on MAD Gaze or Google Glass.
It takes at least three seconds for it
to detect your location and give you the cue.
So we went ahead and used our app
and compared the two of them.
So to test out the demo applications
we tried to replicate a scenario
in a museum for the indoors.
What we did, we had a primary task
which was looking at the points of interest
and a secondary task, navigating,
because we really wanted to focus on multitasking,
rather than on just navigation.
So we had nine points of interest
around the whole five to 10 minutes path,
and for the indoors, we had seven pictures on the left,
seven pictures on the right,
and there was always someone who was walking with the user
to make sure that the user is communicating
and not just looking into the screen all the time.
As well as our path for the indoors
was really variable, with variable lighting settings.
It had staircases so that we are really making sure
that user's not looking into the screen.
So our points of interest for indoor were something like,
we put bunch of pictures of animals, fruits,
or some quirky stuff so that we make sure
the users are more looking into the pictures
than into the screen.
And for outdoors, we had point of interest
something like that in the city walk area.
So we measured four metrics,
three of them are mental demand, physical demand,
and frustration level according to NASA-TLX
from zero which is low and 10 which is high.
We also measure how much time
the user looked into the screen,
and we took a very simple baseline off it,
like low, medium, and often.
And we realized it was very basic
because it was more about the user perception
of how much the person looked into the screen,
and we basically took that time and normalized it
by the total time of the experiment
to just see how much the time user
actually looked into the screen.
And we came up with some, our results with that,
and here the mental demand,
three means that the number of users
who are choosing the mental demand three,
the number of users choosing the mental demand four,
and same like that.
I think this box part explains much better.
So you can see for Google Glass,
indoor using our app, 75% the user reported
a mental demand of less than five.
And same for the MAD Gaze outdoor,
which performed pretty good as compared to the Google Glass,
Google Maps for outdoors and mobile phone for indoors.
We actually had much better results for physical demand
because especially for indoors,
because on staircases it's really difficult
to look into the screen or stare into the small screen
on Google Glass and try walking around.
So we similar did the same results for frustration level,
and we got pretty okay results.
We did a simple t-test and we figure out
that there was a lot of difference
between the two paired-sample data
for both indoors as well as outdoor.
And basically we took the time,
for amount of time looked into the screen
to figure out that the users stare
at least 50% more time for their main activity
using this peripheral vision.
And most of the users agreed that are really benefited
from using such an app.
The two cases we got for color-blindness,
we feel that you can customize the app
according to what color you can see,
but there's a problem with no color detection,
and in that case, we think the person
would rely only on motion perception,
and we're still seeing how we can
use our app for that purposes,
still conducting some experiments on it.
And for strabismus, it was a pretty difficult task.
For both the cases, the users were able
to complete the experiment but with lot of frustration.
So for strabismus, we think maybe having a hardware change,
like having a glass screen on the left,
would have been useful,
but that requires more hardware changes.
We are trying to extend our work for AR and VR,
it's really popular area of interest
in Hong Kong or in Asia,
so we're trying to explore there.
We are enhancing a model with eye-tracking features,
and we are expanding a panel to just see how does it perform
with strabismus and color-blindness cases?
And we're putting in, again, in case you haven't listened
through the entire presentation.
And acknowledgements, thank you so much.
(audience applauding)
- Let's thank the speaker.
Are there any questions?
Well, I have one, because you said you were looking
into expanding it into VR and AR,
so what are the concrete ideas where, you know,
would you kind of have some items to pick up
in augmented reality, or how, what's the scenario
you're currently considering?
- Mostly games, mostly for AR games.
Especially because in lot of games that you wanna show
there's an enemy on the side and you wanna show
to the user that there's something blinking.
It's very nice idea to just showing
through motion perception,
so that's one thing we are looking into,
whether you can use that way.
- [Host] Nice, any more questions?
- [Audience Member] Did you prototype it on Google Glass?
Like, what was the hardware you used
and how long were these study sessions
to kind of see whether it was overwhelming for them or not?
- Yeah, we deployed both on MAD Gaze and Google Glass,
so the app is actually deployed there.
And most of our experiments were
between five to 10 minutes long.
And most of them were first-time users.
We didn't repeat the same experiment
with the same user all the time
because we didn't want the user to know
what the person is getting,
like, what the experiment is about.
And I guess, first of all,
with the entire Google Glass or any smart glass,
the battery lasts for two hours,
so you cannot really run any experiment
more than two hours, the battery will run out.
So that's a limitation on the hardware section
for any smart glasses currently out there.
But our experiment were mostly running max for 10 minutes.
Some users maybe took more than,
especially the color-blind person,
its completed experiment took like 15 minutes
to complete the experiment.
But honestly, if you use any smart glass,
irrespective you're using for this app or anything,
even when I use it for when I was developing the app
and testing it myself, after a half an hour or 40 minutes,
I was completely having a headache,
because that's a limitation of the smart glass,
and that's what we are struggling to,
I guess that's why the Google Glass
faced a lot of criticism in that area.
- [Host] One more question, yes.
- [Audience Member] So, just related to this question,
did you also test the standard Google Maps app
just to make sure, or just to have this control condition
whether this one is performing?
- Yeah, so maybe I didn't explain it or I spoke too fast,
I'm sorry about that.
But we tested out our app with the Google Maps in MAD Gaze.
We picked MAD Gaze, it had a better performance
than the Google Glass.
But we tested out the app actually with the Google Maps,
and the results are, basically the results that I talked
about are actually comparing Google Maps on the,
that MAD Gaze glass navigation app
is the Google Maps that we tested with.
- [Audience Member] You also mentioned that just talking
about augmented reality,
the way Google Glass works
versus future augmented reality experiences,
do you think the findings will hold true
for augmented reality experiences
if the field of view is pretty wide?
- What do you mean?
Like, if the field of view
is much better than-- - It's just pretty wide
and you can have many
other forms of-- - Yeah, like color lens, or?
- [Audience Member] Notification, exactly.
- Yeah, that's what, currently, in my point of view
there's more research done in VR than AR
because AR faces a lot of limitations.
But I think one problem with increasing more field of view
is you will have a bulky headset,
so I think maybe they're trying to solve that issue.
So I guess, yeah, I mean, I'm not sure
if I correctly understand your question,
but if there's more field of view,
then I guess AR would probably be less important
than VR in that case,
but also having more field of view
makes your glasses bulky, so I'm not sure what direction
the industry wanna really take into in that sense.
- [Host] Okay, let's thank the speaker.
- Thank you.
(audience applauding)