we have been looking at
physiological monitoring to try to infer
how hard people are working on a given
task what we've been trying to do is
find
the least intrusive means
for collecting physiological data
we were looking at the aviation domain
but it can obviously be extended to
a lot of other a lot of other areas
so initially we had a lab study which
looked at a very simple computer task
where people were tracking
balls and shooting them and we could
quite easily
varied demand by increasing and
decreasing the number of balls in a
given pattern and then we could see if
the physiological measures reflected
that
that pattern and correlated well with it
so um
that was a lab study to
see how feasible this is and then we
went on to
doing a helicopter simulator study one
of the questions
that we had was
would highly trained individuals and
helicopter pilots
reacts in
much the same way as
an usual person does you know to
increase level of
demand we did a computer file on
something that sounds similar where
horia was using a brain scanner and
turning the lights red when people were
practicing things like air traffic
control is this a similar kind of idea
yes the idea behind it is basically the
same the efner sensor what i was using
is
maybe more accurate in certain
scenarios and reflects more of the brain
activity
we are aiming to use thermal imaging as
a way to make it less intrusive
so people don't have to wear any sort of
equipment so this small flirt thermal
camera the resolution is quite small
it's 640x480
so it's not an amazing resolution there
are high resolution cameras nowadays but
still not
up to the level of phone camera what it
does it's picking up thermal radiation
this should be within the range of
7.5 to i think 13 micrometers
and it's converting it to
temperature
so obviously that's one step
of the way
the other
the next step was extracting temperature
from various areas of the face
i'm hiding behind the camera here this
is the scale here on the right
blue is cold black is even colder
and then as you go towards red yellow
and white that symbolizes a higher
temperature
yeah
when we extracted
the data we used
these sort of images to find facial
features those facial features allowed
us to get data from various areas we
covered quite a large area of the face
whereas previous study looked at
smaller areas like the nose forehead
and indeed actually it
most of the changes occur there on on
the nose and around the nose area people
have reported it before
we try to do a
more structured study where we could
control the demand in a
more accurate
way so what we've seen that usually in
most people but not in all of them those
temperatures
tend to drop when they are engaged in a
high demand task
and actually they do tend to go back up
as the task demand diminishes the nose
tip is the most evident point
but
also the side areas of the nose show a
similar response just not as large here
in grey you can see the perceived demand
so there are three levels of the task
towards the middle of each of the levels
the demand peaks so things get more
difficult yeah things get more difficult
and at the same time you can see here in
green for example which is temperature
in point p which was the tip of the nose
you can see how that drops and then as
the task becomes
easier again it starts going back up
this here in blue is point v which was
right below the nose
so here you can see also drop maybe it's
not as high but still you can see
sort of the same pattern we got a
similar pattern in most people and in a
lot of the helicopter pilots some other
people do have naturally colder noses
they're just cold by default so then
there won't be any
meaningful drop in temperature
uh that's yeah one of the
one of the other effects but
yeah we're still exploring to see if for
those cases
there are other
signs indicating
to the increase in demand
is this something that you can do in
real time or how does it work
we can actually do it in almost real
time
with
the algorithm i used which i have to
admit is not highly optimized but you
could do it maybe one frame per second
the camera itself does not
output a very high frame rate it's
around 7.5 frames per second
it's actually limited
uh
you cannot buy it at higher
frame rates
and
yeah then the videos are quite large
because it's the data is not compressed
as you would get in a normal
visual
video
so it could end up being within the
range of 30 gb for for for one of the
videos and then
yeah there's it's quite a lot of data to
process
initially the video is outputted in a
digital level that the camera
records which is then converted to
actual temperatures by using a
calibration curve and then
we used that
temperature basically matrix
to convert it to a visual rgb using a
color map
which obviously you can
change the format coming out of the
camera is
is digital so it's it's the temperatures
and then you convert that to video do
you ever go back from the video the
visual video back to the original to get
the temperatures uh yes yes i do that
at the end so once you get an area of
interest and you need to extract
data from it
then you go to the original
matrix of temperatures and you get the
temperature area from that
in between those specific coordinates
we're used to seeing a pixel as being
three or four numbers rgb alpha whatever
so instead of having that you've got a
temperature in kelvin or something yeah
you just have a temperature
well i used to convert it in celsius but
yeah you can convert it in
any unit based on that color map
we did the
tracking of features
we had various approaches to it
the first approach from the paper that
was published
relied
on
cascade classifiers to detect the eye
area
so it was trained to
to pick up eye patterns
and to find the eyes and then we would
rely on another property of the thermal
image the pupils are colder than the
surroundings so if you look at this
rectangle which is zoomed in here this
is in actual thermal data so whereas
here you see the rgb picture the height
of this represents temperature so you
could see here how temperature drops
across the pupil so this is basically
the center of the pupil the low
temperature there is most likely because
there's no blood vessels passing through
the pupil then it could quite accurately
find the center of the pupil and then
going from there we found the rest of
the landmarks
for the helicopter study using
this smaller camera we used a different
approach which actually didn't belong to
us we retrained an algorithm that was
previously published
and it's
more robust to poster changes and it
works better and faster we retrained the
algorithm to pick up these specific
points on the on the face it does fail
at
points in extreme angles but then we
filtered out those intervals as you can
see
as
we
go through the video
the nose gets
colder
this pilot was going through a very very
difficult scenario where he
was performing an auto rotation which
means that he had
no more engine power and he was just
going down and trying to cushion the
landing just by using the propellers
that were not spinning at the moment
here as you can see they crashed and you
can see the nose being really really
cold he looked like he laughed then yeah
he laughed yeah
well luckily it was a simulator you can
laugh after
[Laughter]
can you remind me what pethagoro's
theorem
let's is
the triangles
the lights are about five seconds behind
what um max is going through
this much it takes from the heart
to pump up the blood
uh 32-bit images that is um
four channels per pixel
is very common even when we're not used