- Ah.
Okay.
Hi, I'm Crystal Butler from New York University,
I'm gonna talk about miface,
a framework I and my co-authors developed for
defining facial communication mappings
using a 3D avatar and crowd sourced natural language labels.
I'm going to give a little bit more information about
the context of this research and the motivation behind it
because I find we get a lot of questions about that,
so I will get to the actual research a little bit later,
but there's some information first.
First, lets clarify what a facial communication
mapping actually is.
Facial expressions are one modality
of non-verbal communication,
and like words they have both a semantic component
and a syntactic or structural component.
The semantics are meaningful information conveyed
to a viewer, the perceiver of the signal.
The syntax comprises the dynamics of craniofacial
muscle activations, which would be for example,
whoops, that's not what I meant to do, hang on.
So, like this.
Or this, this are muscle movements.
Um,
the image shows expression semantics reduced to
a single natural language label which is here,
a key part of the miface framework
that I'll go into shortly.
Some of you may be familiar with FACS, the gold standard
of facial movement and coding, which provides the basis
for modeling muscle activations in our framework.
FACS defines numbered action units, or AUs, for each
individual muscle involved in producing facial expressions.
AUs can be further described using a discreet
intensity scale ranging from A to E which we transform
to a continuous weighting ranging from zero to one.
FACS is not meant to be a guide to interpreting expressions,
only a tool for systematically describing facial behavior.
However, the FACS investigators guide contains an
emotion prediction table which lists
six basic expressions of emotion
and associated sets of prototypical AU activations.
40 years after the initial publication of FACS
those same six basics, and some people include contempt
as a seventh, are the only generally accepted
facial expression mappings.
Here we have images from the Cohn-Kanade CK plus
facial image expression database,
alongside example miface expressions,
here,
displaying disgust, sadness, fear, surprise,
happiness, and anger, which are the six basics.
The CK plus database is an example of label training data
that can be used in machine learning
for automated facial expression recognition.
CK plus and similar image sets
are FE mappings of AUs to appearances and emotion labels
typically constructed by taking hundreds of post photographs
displayed by human FACS experts
or actors who have been coached by them
which is an expensive and time consuming process.
The requirement to create enough labeled images
to provide a variety per expression
required for machine learning limits the number
of expressions that can be modeled in this way.
Even the smaller expression sets required
for social sciences studies are constrained
by the availability of human and technical resources.
In both fields, emotion labels for images are usually
validated by naive human judges under a forced choice
experimental design in which they select from a
predetermined list of labels, generally determined a priori
by experimenters as the emotion is being displayed.
So for example, I might show you this image
and ask you whether it displays fear, anger,
happiness, disgust, sadness, or surprise,
those are your only choices.
Facial expressions aren't just the concerns of
machine learning specialists who want to use
facial feedback to sell us more soda pop
or psychologists researching emotion,
they effect all of us.
Around 65% of communication that occurs
during human interaction is non-verbal.
While we also use gestures, tone of voice,
or posture to convey meaning, the face is our primary focus.
Faces are so important that we have a specialized area
of the brain for recognizing faces, the fusiform face area.
And facial expressions can convey more than emotion.
They can convey mental states such as interest or boredom,
physical states such as pain, moods, social emblems such as
politeness, information about individual identity,
the current US president is a great example of that,
and action tendency.
My favorite example of an action tendency which I believe is
a major oversight that exists in facial expression mappings
is a combination of AUs 18, the lip pucker,
and AU 22 which is lip funneler,
and you put them together and you get the kiss.
So, humans send and receive facial expressions
as social and emotional signals.
Understanding the way humans recognize and display
these signals can lead to improved diagnostics in medicine
and psychology as well as treatment tools
in the face of conditions such as ASD or stroke.
Shown here is a list of some conditions in which impaired
facial expression production or recognition are correlated.
With recent advances in machine learning
computers are able to perform tasks
we once believed only humans to be capable of.
Facial expression recognition is one example.
Here are several companies who have production software
that performs automated FER for tasks such as
market research and usability studies.
Two of the most well known, Affectiva and FaceReader
can recognize individual AUs
but still only return metrics on seven emotions
because there's no way to map the AU space
onto a broader expression semantics space.
As an aside, Emotient founded in part by Paul Ekman,
who I showed earlier, was recently acquired by Apple.
The flip side of recognition is expression production,
which virtual humans are becoming increasingly adept at.
Virtual humans can be found in characters such as in games,
or shown here, digital assistants, Amelia here
is a digital IT agent,
Molly is a nurse who lives in an app
that can get feedback from patients to give to doctors,
and at the bottom we have a academic research project
which is a psychiatrist who can diagnose conditions
such as alcoholism and major depressive disorder.
Their facial expressions may be created by using
techniques from animation such as motion capture
and retargeting, or by manipulating the FACS informed
character rigs such as skeletons or morph targets,
morphs are digitally modeled target movements.
Currently the digital facial expression space
is much larger than the recognition space,
but the choice of how facial display semantics
are represented is largely subjective.
Creators are free to represent meaning with any
facial displays they or their actors deem appropriate.
All these avatars are displaying smiles,
the most reliably recognized expressions across cultures.
Is Amelia exhibiting a confident smile?
Is Molly exhibiting a polite smile?
And is our psychiatrist showing an interested smile?
Would those variations be appropriate in all contexts?
We need to build facial expression lexicons
that provide a ground truth of shared
meaning so our digital counterparts can communicate clearly
and understand the nuances of human behavior.
So, we propose the miface framework
as a mean to build such lexicons.
Miface uses a FACS based virtual human
to display expressions and allows naive crowd sourced judges
to provide natural language labels
for perceived signal values of those expressions.
That makes miface reproducible across an infinite range
of avatar appearances that can vary
in age, race, and gender for example.
No FACS experts are required to generate image sets
or motion and graphics, and developers are free
to express the entire range of movements the human face
is capable of in any combination in a way
that is consistent across facial morphologies.
So, to prototype the framework.
We first needed to build a FACS-based avatar
and computationally generate facial expressions.
After testing several alternatives,
we decided to use Faceshift, also since acquired by Apple,
as the primary tool for avatar creation.
Faceshift provides a basic polygonal mesh
and repertoire of face movements, or morphs,
informed by FACS.
By acquiring a depth map of a target actor
using a connect sensor, the base mesh
was transformed to the targets geometry.
In addition, Faceshift allows for modeling custom morphs.
The generated mesh and customized morphs
were exported to Maya for hand modeling adjustments
and to design any missing morphs.
Morph activations were rated on a scale from zero to one
and finally the avatar prototype was skinned
with a low resolution texture.
The resulting avatar was capable of 28 FACS movements.
So with a potentially infinite number of
movement combinations and intensities available
we decided to run two proof of concept observer
judgment studies with a constrained set of AUs and weights.
The studies used crowd sourced naive judges
to decide whether expressions were believable,
and if so, to label them.
22 key AUs were selected by combining the guidelines
for identifying basic expressions of emotions
as outlined in the FACS investigators guide,
and combined with criterion coding of expressions
from the FACS certification test.
Weights were fixed at levels at which
they were clearly discernible.
All combinations of three and two AUs were computationally
generated based on the selections I just showed,
and used respectively in two proof of concept
expression labeling studies.
This grid shows a small sample of triples,
or combinations of three.
Some generated images had to be removed prior to testing
because additive effects of particular AUs at these weights
were, resulted in impossible configurations.
So, next step: human judgment studies.
We crowd sourced participants from Amazon's mechanical turk.
Images were sent for phase one testing in two by three grids
for binary classification by three
mechanical turk workers each.
Workers checked a box to indicate yes if they believed
the expression could be displayed naturally by a human.
If two or three raters said yes, the expression went on
to phase two for labeling.
This figure shows an example image grid.
So for study one phase one, which used three action units,
we began with 1,380 images and 341
ended up being selected as recognizable.
Study two phase one we used two AUs,
out of 229 candidate images,
114 passed on to the second phase for labeling.
In phase two, 40 mechanical turk workers
each provided the single word label
they felt best described a facial expression.
This figure shows the instructions given to participants,
showing an expressive face here,
to be labeled along with the neutral expression
which is an expression free face,
and that was provided as comparison.
Each target image was shown in this format.
The next step in our process was finding a way
to reduce a set of free response labels
to a single representative label.
This was absolutely the most
challenging part of our project.
We could find no standardized quantitative measure
of word-word synonymy, which is what we required
to analyze our label sets.
We tried several algorithms that used the Prinston
word net ontology, but none provided consistent results.
We then tested two natural language processing algorithms,
were Deveck and GloVe.
GloVe performed better over our input domain.
To calculate similarity scores every label is paired
from all others within a set for N times N
minus one over two pairs.
In the case of 40 labels that gives us 720 set.
A cosign similarity score from minus one to one
was the sign using pre-trained GloVe word vectors,
with minus one indicating opposites,
and one, an exact match.
The scores are summed and the top scoring word
became the expression label candidate.
Before deciding the top scoring candidate label
was our set representative, we wanted to test
overall semantic coherence of the set.
In other words, we wanted to ensure that the label
represented a single cluster of strongly related words.
We decided hierarchical agglomerative clustering
was best for our analysis because it generates an
arbitrary number of clusters based on distance measures,
our cosign similarity scores.
As shown in this figure, a dendrogram,
clustering starts with individual words
at the leaf level down here
and joins them as you move up based on
how closely related they are.
All clusters eventually join here at the top.
So in order to get a meaningful clustering result
you need to provide a cut-off.
Expert testing on a sample of label test determined
that a cut-off of .8375, which is here,
combined with a requirement of greater than 75%
set membership in the primary cut cluster
gives a good indication of coherence.
So, in that case, what we have here
is this is our primary cluster right here,
and these get cut out.
So, in study one, our three AU study, phase two,
157 of
341 label sets passed clustering,
and study two, the two AU study in phase two,
63 of 114 sets passed.
So, after beginning with 1,609 candidate images
we finally got 220 expressions reliably mapped.
A few examples are shown here
with their associated AU weightings,
we've got embarrassed, curious, and hopeful.
To the right is a table of 51 unique labels
that were returned across both studies.
As you can infer from the ratio of expressions to
unique labels, image label mappings
are not necessarily one to one.
There's plenty of future work to be done in our framework,
such as improving avatar quality and word synonymy scoring.
None the less, we expanded the expression mapping space
by about a factor of eight
with these two proof of concept studies alone.
User interfaces that can understand and model
social and emotional behavior are critical
to advancing the supporting role
technology plays in our lives.
We believe miface can be used to develop a common ground
of facial expression production and recognition
that we hope will lead to improvements in areas such as
game character design, the development
of the intelligent agents, machine learning,
and psychological testing and treatment.
So thank you very much for your attention,
and I'll take any questions.
(crowd applauds)
- [Host] Thank you very much, please step up to the mic.
Don't be shy.
- [Man] So I have a question,
so you used a particular piece of software
which sadly is no longer available to regular folks
to use to create the faces,
what difference do you think there would have been, if any,
if you had used one of the other pieces of software
to actually create the avatar expressions?
- The goal is to get the most realistic
movements possible
with a secondary goal of a realistic appearance,
although that is not necessarily required,
you don't need to have hyper realism for people
to interpret expressions, so if there were something
out there that we could've gotten a better
movement modeling results or better morph targets out of,
that would've been great, I can't say that this prototype
is actually perfect in that sense,
that's something that we're still working on.
- [Man] Okay.
- Does that answer your question?
- [Man] Yeah.
- [Host] Anyone else?
Don't make me be the only person asking questions,
come on, this is your chance to actually talk with
and ask questions of the speaker,
you won't be able to ask questions
of the ACM digital library as easily.
No one else wants to take me up on this?