Transcriber: Mark Freehoff Reviewer: Denise RQ
Thank you. It's a pleasure to be here.
When something really matters
it elicits emotion
and how do we help people communicate emotion,
to make life better, whether at home,
at work, for your customers, for your kids at school?
Our challenge at The MIT Media Lab and at Affectiva,
has been to come up with better ways
to measure and help people communicate emotion.
I'll start with an example that we did years ago
which was a skin conducting sensor
that we gave to an audience like you, worn on the palm of the hand
and it glowed brightly every time
the audience got excited. Every time they were engaged.
We learned a lot that day.
We learned for example that every time a new speaker
came out on stage, the audience glowed.
Every time there was a live demonstration,
whether it worked or not, the audience glowed.
Every time there was live Q&A or laughter,
the audience glowed but unfortunately, every single time
there was a PowerPoint presentation
there was a decaying exponential in brightness.
(Laughter)
Let's, let's look more closely
at an example of the signal changing.
Here, for a nine year old boy, watching a movie trailer,
the signal will go up about a second after each of the parts of the trailer
that cause him to engage.
Starting with the music.
(Music)
The three biggest peaks.
(Laughter)
See this is where it kind of falls of here though.
(Music)
He peaks for the name of the movie and the date of it,
so you can see why people in Market Research and Advertising
are extremely interested in this information
or I as an educator, I'm extremely interested too.
'Cause I want to know what content
is connecting with my students and with my loved ones.
Here's an example of the signal changing with a Patriots fan,
back when the Patriots were in the Superbowl.
And this one illustrates an important point about skin conductance
and that is even as we measure the auto-immunic arousal
of a person and their excitement.
This goes up with things that are good or bad.
So we see it go up here when the guy has a --
when his favorite team scores a touchdown,
it also goes up during an endzone overthrow.
So it goes up with good and bad.
Interestingly, it also goes up during a Dorito's Mouse T.V. commercial,
when the mouse comes out and catches the man.
(Laughter)
It's interesting now that we can get this information in real life,
to see how relatively important different events are in life
and where the significance of something that you share with somebody,
compares with this response to some other things that are going on with their life.
What drives this signal?
This skin conductance is driven by the autonomic nervous system,
sympathetic nervous system response.
Your autonomic nervous system has two main branches,
the sympathetic and the parasympathetic.
The sympathetic can be thought of kind of like stepping on the gas, in the car,
it revs your heart up, it causes the skin conductance to increase.
The parasympathetic causes the heart to decrease.
The sympathetic is thought of as fight or flight,
the parasympathetic is rest & digest.
Almost all the organs are innervated by both the sympathetic and the parasympathetic,
except the skin, which gets the pure sympathetic,
which makes it ideal for us measuring this fight or flight response.
Now, I became interested in measuring this response when I started working a lot
with people in the autism spectrum, who are non-speaking.
The most heart-breaking stories, that I hear from these --
from the families and the individuals themselves,
when they later acquire the ability to speak,
is that, the changes that were worst, were when they got kicked out of school
or were shipped off to an institution,
removed from learning opportunities and social opportunities.
What caused that to happen?
Usually it was a meltdown or a series of meltdowns
and the meltdowns, like adult tantrums, appear to come from nowhere.
One moment the person seemed calm and relaxed,
the next moment they might have become injurious
to themselves or to others.
So, they seemed very unpredictable and uncontrollable.
Now, as we undertook the privilege of getting to know some individuals
who could later communicate, they told us that these meltdowns
never came from nowhere that, in fact, they were always preceeded
by increasing stress and frustration.
They couldn't understand why people couldn't see what was bugging them.
So this made me think, "Wow, what if we could enable them
even though they can't communicate,
to signal out their increasing stress and frustration?"
But to do that required a device that is shown on the left here,
a classic, galvanic skin response,
skin conductance sympathetic nervous system measure,
that involved a lot of wires and boxes,
and that was uncomfortable to wear in daily life.
So my team at MIT went through lots of iterations
until we came up with a device that could be worn robustly
on the wrist and gather data in real life.
This has just been commercialized by Affectiva
and I'm pleased to say is finally available as a product
for people to gather and communicate this information
in the real world, outside the lab.
Here's the first example of seven days of data that I saw from a student at MIT
wearing the skin conductance sensor on his wrist.
And the first thing that jumped out here was there's a lot of studying and it's very activating.
Studying and lab work, in fact, sort of to the discouragement of we, professors,
is much more arousing that any of the classroom activity. (Laughter)
We also saw these huge peaks that were happening during sleep which puzzled us in the first place.
We just expected sleep to be flat.
Generally, as we did further polysonography,
and teamed up with some expert sleep researchers on this,
we found that these peaks tend to occur
during slow wave sleep or deep sleep, and we're now looking at the relation
of those to sleep quality and memory and learning.
Here's an example, of a girl on the autism spectrum,
where she's able to communicate out her arousal level, using this sensor.
Here she's wearing one on each ankle so you'll see two signals streaming
across the bottom.
She's wearing it while she's undergoing an ordinary occupational therapy session.
This is 45 minutes of an ordinary session here,
and you'll see, you know, it peaked here
and she had a little bit of a meltdown here, got in a ball-pit,
calmed down, peaked here, calmed down.
What you see up here, in blue, is right now, this window here
the right edge is what you're seeing in the video and we see that it peeked
as she climbed on this swing, but as she starts swinging,
as that rhythmic activity kicks in, it has this nice, decaying exponential.
We see for many people on the autisim spectrum, when they rock, or do these repetitive movements
they can be very calming for them.
They're not just doing that to be difficult or something.
It didn't have a very important, regulatory effect.
In fact, I've noticed an increasing number of rocking chairs, cropping up in airports
since 9/11 and they're almost always taken, at least at Boston Logan.
I think that's a healthy sign.
Once you get the ability to make a new measure in the real world
and have people wear it, unobtrusively in daily life,
you learn a lot of things that are surprising and this is a really special surprise we ran into.
One of our kids on the autism spectrum was wearing this over Christmas.
I was looking at the data and I saw this enormous peak on one side
not the other side, I thought: "Oh dear, something is going wrong with the sensor!"
I looked more closely and it looked like it was fine before the peak and after.
I asked his brother, "Any idea what could have happened here?"
and he said, "I was with him and he had a seizure right after."
Wow! I got on the phone with Joe Madson, Boston Children's Hospital,
"Is there any way somebody could have a huge auto-immunic surge
on one side of the body with a big seizure?"
He said: "Yes"
Fast forward about 8,000 hours of data and extensive analysis,
led by my student, Ming-Zher Poh for his doctoral work
and we have now shown that measuring signals from the wrist, reflects signals
measured from the brain and the seizures are labeled
by the epileptologists reading the EEG only, they coincide exactly with these peaks,
measured just from this electrical signal, from the surface of the skin, on the wrist.
So this was really mind blowing to me!
That a change in our brain could show up as these events, picked up through a simple wrist band.
We're hoping this now can lead to better detection and treatment in real time
of people who suffer from seizures, of the seizure events themselves.
Now another thing that kind of surprised me in doing this, was --
the events show up, sometimes, on just one side.
Usually the two sides are showing the same information.
We where just measuring two sides for redundancy, to cancel noise,
but we found that -- we learned something that was already in the literature
the right hemisphere of the brain controls the electrodermal activity of the right palmar surface,
and the left, the left.
I sort of rediscovered this when a tragic event happened in my family.
I lost a loved one and that day I was at work, having trouble speaking.
Those who know me, know I never have trouble speaking.
So, one of my colleagues, Ron Elco, said:
"Put a sensor on both sides!"
I'm strongly right-handed and sure enough, the left side was incredibly suppressed.
That night I was, sitting in the kitchen, in this total stupor,
just kind a stirring a pot of something
and one of my sons came up and started rubbing my back
which, I felt really blessed, this is a very special moment,
I almost felt hope restored and then, his little brother came up and did the same thing to him.
I had one of those rare moments of parental bliss, where you kind of tingle down to your toes
especially as a mom of sons.
And -- (Laughter)
at that moment, my little one looks up at the display in the kitchen, 'cause of course
I'm streaming my data wirelessly to this monitor,
(Laughter)
and he says: "Mommy, you're lines crossed!"
My right signal that was way above my left, had come together
during that moment of parental bliss.
Now I thought: "Wow! I've heard about positive and negative stuff for the brain
you now might show up but I never thought this could show up on the wrist.
Now, that's just a few data points, but I monitored myself overtime,
with sleep and sure enough, the signal, as I recovered,
went back to the normal sort of switching, left and right.
Now this is just a few data points but it shows that there's this whole new potential to explore
what's going on in the brain, even by measuring some of these peripheral signals downstream.
There are many positive moments as well, that people have shared with us,
using our emotion technology.
Here's one from a bride in India, who, naturaly, was sort of stressed out before her wedding
you see lots of peaks.
Her friend loved seeing this part where she was chilling with her friends
and they helped calm her down before and then the four major peaks here,
during the wedding ceremony.
The first, where the vows were exchanged,
the second, third and fourth in the Hindu ceremony
where the bride and groom exchange symbolic gestures
that they are now husband and wife.
Here's an example where a mother and a daughter wanted to share
the daughter's experience at school.
The daughter had her first, big concert, in front of school that day.
She was playing a harp and singing in front of the entire school
and she expected it would be kind of peeked there, and it was.
Also it was very peeked during physical education, to be expected
anything that gives physical exertion, makes you sweat, can be very arousing.
Her mom was very disappointed to see that math lessons,
(Laughter)
was the low point of the day.
But there was this other, huge peak there between Reading and PE, and --
What was that?
Turned out the daughter had a talk with her mom, later that day.
In fact as she recalled this experience, we see the emotions coming back up again.
This was this horrible, bullying event that she was being dragged into by her friends.
Wouldn't it be amazing if the people we care most about
were given the opportunity to share, in an externalized way
some of what was going in their day that was so difficult.
Whether they can speak or not, this can be a powerful way to take your emotions outside of you
and begin to reflect on them and share them with others.
I have one last example with the skin conductance sensor, a personal one.
I discovered one day while taking my son
ona birthday outing to Six Flags, that I indeed had big peaks
with the Roller Coasters, which I love,
and a really awful one with a ride that I won't tell you about, that was very unpleasant.
(Laughter)
But the surprise to me, when I looked at the data at the end of the day,
was not that I had this huge peak when I'm on the fastest, highest, most intense roller coaster
in all of New England, but it was actually earlier in the day
when we were just trying to get out the door.
(Laughter)
Many people on the autism spectrum who are conversant,
tell us that they have difficulty reading facial expressions,
they are trying to pay attention
to what you're saying and at the same time,
they can't tell if you're looking pleased or displeased, interested or bored.
In particular, many of them have asked us if we could please help them
to have some technology that tells them if the person
they're speaking with is still interested or if they're boring them.
So this next example is a short video clip, introducing my collaborator on this work,
Dr. Rana el Kaliouby, who' s also my co-founder at Afaktiva
and the chief architect of our face reading software.
"-- Which would be recording your facial expressions and analysing them in real time.
So we use 24 feature points, on the face, like the the eyes, mouth, eyebrows,
and we use those to recognize facial expressions like, a smile, a lip pucker,
eyebrow raise and a mouth open and so on.
And we look at these over time and we sort of use them
along with head gestures to make a guess at your hidden understatement.
(Video)
Each person is wearing a camera, pointing at their own face often.
(Video)
(Laughter)
All right. Using this technology, there are many things we can recognize
and many things we can learn, for example I learned that actually, I was quite interested
in what he had to say, but I started to feel dis-interested
because he never paused,
he never gave me a chance to go into thinking mode,
process the information and return.
Or when I did, he had moved on and I was lost.
So as we do this fine grain measurement of emotion, we learn a lot of things about how to improve
our interactions with one another or how to improve our interactions on-line
or elsewhere in the world.
Here is just an example reminding us that, you know, while emotion is about what matters most,
it's also communicating this complicated language, that is non-verbal.
In order to get a computer to learn how to read this,
we have to solve an extraordinary, complex problem
that makes chess and Go look trivial in comparison.
With over ten thousand facial expressions that can be communicated and changed,
every second, multiplied by the faces participating in a conversation,
the social, emotional games people play is probably the hardest computer science
problem to solve, in the next century as we go forth.
The complexity is just astronomical.
What we are starting to work on at MIT and in partnership with Affectiva, in this next project,
is a way to go online, to get right now thousands and hopefully, before long,
millions of examples of natural facial expressions, while they happen online.
Here's an example where we partnered with an interactive task on Forbes.
You can go to Forbes.com and try this out.
You click on one of the T.V. commercials. These are designed to elicit delight
and we measure your smiles and we show the results here
for one of the commercials, for three groups of people.
Those who say they have seen the video many times,
those who have seen it once or twice and those who say this is their first time.
This should be playing.
But for some reason it's not playing. Clicking!
And what will happen when they get there and play this video
is you'll see the video synchronized with the peaks, on the graph.
You can go online and see this, if they can't get it working here.
If you go to Forbes.com and search for interactive smile
you can participate, give our researchers your facial expression data.
You can also see the results and see interesting things, like, you know,
"Do people smile where you smile, if you smile at the dog?"
One thing we learned is that people who have seen it before
smile much earlier in anticipation and they also smiled a lot more,
for these popular videos that they like.
I seem to have lost my slides completely here. I will wrap up and just tell you
whether the emotions are online or in real life,
whether they're to help somebody who is speaking or non-speaking
or just to capture some of those most positive emotions,
moments in life, new technology,
is enabling people to communicate emotion, in ways never before possible.
I'd love to hear from you, how this technology could help improve lives for you,
your business, your customers and your loved ones.
Thank you.
(Applause)