This is a virtual human
She's currently not really paying attention to me
but but later on we will turn her on and then she will be able to
respond to me when I said virtually even I really what I mean is an embodied conversational agent, but that's quite a mouthful so
we just call it's shorthand virtually human the point is that
it looks like a human but it's on a screen so it's not a robot and
It behaves and talks and listens the way humans do
We use two sensors really so we use a microphone and a camera so she can see me
basically, it's her vision and she can hear me and
We can begin give the trials. Hello
What are we doing today?
Yeah, sure
For each question, please answer in terms of options to show them the screen
More than four days every day. Okay. Here's the first question over the last two weeks
Not at all
Yes
Okay, here's the second question
Tower Bridge
You answered no
I'm sorry. Can you please repeat your answer?
Brown sauce
No
It's a study to see if we can replace a written questionnaire with a with a virtual human questionnaire, so that's it's a bit different
the mock normally do we also made a
Version of this called Alice where you can ask things about the book Alice in Wonderland's and in general
There's many different ways in which you can use cultural units. I suppose all you're seeing is a unity
skin a unity character and you can replace it with any unity character that you want but in it all starts with
The input basically she you can listen to what I'm saying
And you can see my face. So we're basically creating characters that are
linguistically skilled
emotionally skilled socially skilled
So they there's a speech recognition module in there
That basically captures all the words that I'm saying and based on that
She can determine what how to respond, but she also takes into account
My face and my facial expressions, so that means she knows when I'm looking at her when I'm looking at her it means that
She knows when I'm smiling not smiling and then you can start doing interesting things
Because then the agent can start back channeling so it can also start smiling back when I smile
Nodding when I not or if you want to create a particular
Persona, a particular character that is absolutely abrasive or even aggressive you can start doing the opposite
So we made a character called spike
Always an answer for everything and if you not he will shake if you say yes, you'll say no
And people find it really engaging but they get really worked up talking to spike. So there's the
Let's say the sensing component that recognizes your face your facial expressions your voice your speech. Also how old you are and
that
Information gets sent into what we call the dialogue manager and the dialogue manager maintains
What to say next and also maintains what's you know what we call dialogue state?
So what has it already said?
what I said topic that we're talking about at the moment and then based on the input and the current state it decides what to
say next that is sent in a very high-level commands using something called behavior markup language to the
Realizing and they realized it and turns that all into the actions that need to be made and the speech that needs to be
Spoken quite a lot of effort went into this particular virtual agent into making sure that there was synchronized lips
the speech and
That you can interrupt the agent because in the past quite often people made these virtually humans were in a sense. They played a little
Little pre-programmed movie. It was a bit of code
of what to say that was turned into a set of behaviors and
Once that was executed they were played and there's no way of stopping it
But of course in real life you quite often want to interrupt the agent, but then you need to be able to stop that
behavior realization and
Gracefully going to a neutral mode into a listening mode, etc
So that's the kind of developments that we're working on with these virtual humans
Is this some kind of machine learning or is it what's going on behind?
No
It's it's not a machine learning but it's an algorithm that can go from any
State that it's in back to a neutral state to the next day. So it's it's an algorithm that allows you to break
Where you currently are and go to a next phase while it's running
Rather than having to wait until the end of the segment to play and then going back
what sorts of challenges do you get from having to recognize speech and you know getting a device or
a virtual human to respond
yeah, so there's actually a number of components that they can do if you might have realized that having before the speech recognition happens, so
The first one is is somebody speaking or not. So this voice Activia detection
and then another very important one is
Turn-taking if the agent starts interrupting you while you're speaking that's really annoying
So you need to know when somebody is done speaking and you can't just work by
Voice activity detection because sometimes there's little pauses in my speech and you're not supposed to just button
So
So that that's actually done with machine learning where we learn exactly when it's okay for an agent to interrupt
Not interrupt to take that turn
although for some agents
Sometimes it's it is
Appropriate to interrupt but but at least when you know when when somebody is speaking their turn
Then the agent knows that if they would speak now, they would be interrupting and that would of course have a particular
Effect only conversation the person might be unhappy about that
But maybe that's the the point of the interruption as part of this project and other projects we have recorded many databases
quite often of people talking
Human to human but through screens. So it's as realistic as possible to the setting as we would have with a virtual human
And then we annotate all the data for when I was speaking. What are they saying? Are they smiling or not?
And and of course then from that also deterrence is whose turn is it and then who's you know?
When those go into in somebody else's turn and then you can also when was there an interruption and how was it dealt with?
And yeah, so there's a lot of there's a lot of manual work going in to that
We can show you some of the these are our dashboard in a sense arousal is an emotion. That's basically how
Excited I am and that was relatively high
I don't think it's currently doing valence because this is from video and my face is not currently in front of the camera
But it would recognize whether I'm happy or not happy and this is where I was a child youth and adult or senior
Unfortunately, I'm slowly moving towards the senior a bit
And this is some things about what kind of vocalizations do I have MS island. Is it a filler?
Is it a brief or anything else if I swell the valence should go up but I can't
So that's the kind of facial expressions that we're that we're making and then we can work with that
Is there a big database behind here is an ontology. How's it work? There are?
okay, so all the sensing is done on a
On a learning basis, so that's all machine learning
So the facial expressions are learned using large databases the speech recognition. I learned age estimation
emotional technicians all done by by machine learning the
dialogue manager actually is
It's a rule based. So it's handcrafted
So that actually gives you the ability to author a scenario quite quickly
the downside of
doing
natural language processing
With a machine learning approach is that if you wanted to create any new scenario
You would basically need to first collect lots and lots of data to then ultimately
Create what it is that you want to dialogue to be about
and because we are mostly working on these virtual humans as
Tools to do, you know a particular study with or ask particular questions?
or provide particularly information you want to be able to craft to author those interactions really well, and then actually a
Rule-based system is much better with a particular ontology of topics
turns
etcetera sentences and that you can you can all create that of course where we want to go to is sort of a
halfway house between a machine learn and a
crafted version so that we can still craft the outline of the of the
Discourse so the topics the questions to be asked etc. But we want to
Learn or all the possible variations in which you can say that and all the possible variations in which people can answer questions
because at the moment we have to hard code that and that is
Makes it fragile when you were making kind of really crazy random answers to what she said instead of reacting to those
She just realized that wasn't what she was looking. She has no idea what HP brown sauce is so she can't talk about that
Yes, so you you'd want to be able to use a machine learning system?
so she's so she knows it's off topic and that we're not talking about that and then she can go back to what we
Programmed her to to talk about we're basically
handcrafting
everything that it can say and
Yeah, we want to keep that control quite quite tight
Does that give you a reduced flexibility though or so you you putting this in very specific?
Sort of silos. Yeah, so your your
We're not creating virtual humans that can talk about anything and I think to be honest
We're still quite a long way off from that and when I say we I'm talking about the big companies, etc
so the problem with
You know
commercial products like Siri, Cortana
Google assistant is that they don't they're not conversational agents. They are more like voice
commands
Interfaces and that works quite well
But they can't you can't hold a general conversation with them either
Right and I think doing that is still very much science fiction
That's quite a long way away in the artificial intelligence
roadmap
The other thing that's quite different from what we're doing. Is that those voice?
Assistants, they're all voice only
their voice user interfaces
What we're adding to that is your gestures your facial expressions
Not just from the user because it's the first human can sense me
But also she will use her own facial expressions in her own gestures
To communicate with me and that also creates a sort of a point of contact
so, you know, you will be facing her usually which
also allows her to
Distinguish whether I'm talking addressing her or my addressing someone else in the room, so that's a whole
new
area, I think that will seem much more about somewhere in this space when you put my face in I'll appear and when you put
Steve's face in it'll appear somewhere else and this actually solves a really nice problem, right? It's called the one-shot learning problem
How do we convince a phone to let me in having only seen one ever picture of my face?
Which is when I first, you know
Calibrated it the first time and the answer is we don't train a neural network