- We spent 60 years trying to figure out
how to talk more like a computer.
We're reversing that paradigm.
- What is human intelligence?
What does it mean to be human?
What we're building in the lab,
a simulated model of a brain is incredible.
We are defining the future.
- Hello.
- Hi Werner.
I'm Austin.
I hear you've been getting to know us at Soul Machines.
- Our planets and our civilizations
are changing faster than ever before.
Join me as I travel the globe talking
to startup founders using cloud technologies
to make our world more interesting, accessible and livable.
These are the entrepreneurs that are creating
the future we will live in.
This is Now Go Build.
Our relationships with machines
are constantly evolving.
These days, we interact with digital interfaces
more than with each other.
Perhaps it's time to make our digital relationships more
human.
I'm in Hollywood where technological advances
within the movie industry have helped turn our dreams
into reality for over a century.
Computer generated imagery, or CGI, is no exception.
Since the 1970s,
Hollywood has used CGI to create new characters
for audiences around the world to empathize,
connect, and fall in love with.
But how can the achievements
in this technology grow from entertainment to daily utility?
Could cloud-based technologies help create a more
natural dialogue with machines?
Could AWS be the bridge
between our digital and physical worlds?
That's where Soul Machines enters the stage.
With a rich history bringing CGI characters to life,
their technology may be the innovation
that alters the relationship
between humans and machines forever.
I spoke with their CEO, Greg,
to learn more about their digital people.
So tell me a bit about what's the background.
How did you get here?
- So when we started,
what we really wanted to do was we wanted to think about,
how could we use digital characters to deliver
and create services that we can't do today?
If we are spending more and more of our lives living
in digital worlds, how could we experience those worlds
in a completely different way?
- But it's not just about building a product.
You guys are doing fundamental research, basically,
in this space, because this has never been built before.
- As an artificial intelligence company,
we think of ourself as doing deep tech research.
Artificial intelligence is not a product in itself.
It's a bucket of a whole bunch of different technologies.
But the field that we've really focused on being experts
is in the area of cognitive modeling and embodied cognition.
Which, you know, means we are thinking about the way
in which artificial intelligence as a concept can be applied
to create autonomously animated digital characters.
If you are wanting to establish an emotional connection,
as humans we are biologically wired to do that
through face-to-face interaction.
Using an autonomously animated digital person
really enables you to develop the connection
and create a different type of experience.
One of the things that comes up over and over again is,
digital characters are like robots
and the robots are coming for us.
They're gonna steal our jobs.
But if we stop and think about it,
we can provide and we can augment
our teachers and our nurses,
democratize the way in which these services are delivered
to everybody everywhere.
- Yeah, it's not necessarily replacing the workers.
- No.
It's an augmentation of
- It's augmentation and amplification.
When we're learning in a classroom environment,
there's always the fear of judgment.
Imagine a world where you have a digital language tutor.
They don't really care
whether you do the same lesson 20 times or 100 times.
There's two parts to what we do.
So the first is, how do you create a digital person?
We've actually created a tool set we call
Digital DNA Studio,
which effectively gives our end users the ability
to build their own digital person.
You can then design their personality,
integrate voice technology.
What accent do you want them to speak?
What languages do you want to speak?
And then you get into what skills do you want them to have.
The other part of the product is the part that exists
in the cloud, the platform that literally
enables us to bring these characters to life.
We are rendering really, really high quality CGI characters.
We're animating them in real time,
and we are creating as human-like conversations
as we can possibly create.
So having that cloud computing power
wherever the customer is, wherever the consumer could be,
becomes really, really important.
- To create a digital person,
you first need a digital brain.
With a background in humanoid robotics,
their VP of Special Projects, Holly, has further insight
into this groundbreaking technology.
- Humanoid robotics is an approach to, I would say,
artificial general intelligence where you infer
that embodiment is a very important feature
of human understanding or human-like intelligence.
One of the reasons why I got into the field of robotics,
and specifically humanoid robotics,
I was asking this question, what does it mean to be human?
What is human intelligence in the robot lab?
And I happened across this company, Soul Machines.
They were very focused on this concept
of human machine cooperation.
They had this thing they were building,
a simulated digital brain.
Oh my god, this is incredible.
The fact that it was a software based system
meant that more people could interact.
And so the scalability of that via the cloud
or even in the deployment context to me was very appealing.
- So tell me a bit about the autonomous part
in all of this.
The digital brain.
- Autonomous animation is made possible
by building this digital brain
and simulating a digital human mind with a limbic system,
with an emotional engine that allows our digital people
to sense and perceive what's going on in their environment
and react and animate accordingly.
And this is based on perception data in and action data out.
- What is the kind of input
that these digital persons receive?
- We do leverage audio data, video data, text data.
We actually leverage webcam,
and that's really important for our digital brain, right?
So based on a video feed that I'm seeing,
I can begin to detect faces.
Is that user smiling?
Are they angry at me?
And then based on those inputs
there is some scripted actions.
- And why is that more important
than let's say, chatbot?
- What our digital people provide
is this return on empathy.
This human-like connection.
It's more than putting a face on a chatbot system.
Embodiment is a very key focus of our product
made possible by building hyper-realistic
3D models of digital people.
It only works, and it only looks right,
when you can animate those individual digital muscles.
- As Soul Machines works to create
a more empathetic relationship with digital humans,
the intersection of realism with animation becomes vital.
But how do they create their realistic models?
And how does the cloud play a role in making autonomous
interactions feel natural?
I met with their CTO, Glenn,
and their VP of Digital DNA, Colin,
who both play essential roles
in bringing these characters to life.
In my conversation with Greg, he said,
Soul Machines in reality is a technology company.
Tell me a bit how important that is.
- In my mind, it's
making the human to machine collaboration
more natural.
- It has a lot to do with blending
of the physicality of the actual digital person.
How do we create a novel, unique person
who doesn't exist in real life?
- Think about DNA as being the basic building blocks
of any biological creature.
It's a similar concept here.
You take the real human, the real model,
and you break it down into parts.
- What should I think about
when I think about components?
- We call them phenotypes.
It's a person that we've scanned
and essentially what we try to do is we break down
every physiological muscle contraction
that their face can actually do.
And it's a series of artists
who are actually annotating machine learning data for us.
So every freckle, every corner of the jaw.
- What's the result of that?
Is that sort of a five terabyte data set?
- Yes.
- Yes it is.
- So we're here at Otoy,
the only company in the world that is actually
licensed to utilize this technology.
- Okay.
- When you're scanning skin, one of the challenges
that you have with it is that it's a translucent material.
So the technology of this scanner does some very
cool structured light tricks
that allow us to separate the shininess factors
away from the illumination factors.
All the lights are shined in different ways
in very specific patterns
to be able to generate the data.
And then that is run through a program
that builds us a raw scan
which is a several million polygon scan,
with about 20 channels of data.
- And then that goes
into your AI computational engines
and things like that.
- That's correct, yes.
- And this normalization process is super important
to autonomous animation working, right?
We want those gestures,
those individual gestures that are generated
in real time interactions to work perfectly
across every single different blended face shape.
- So you have a background in gaming slash movies?
- Yes, I had specialized
in a lot of character work in the film space.
As hardware got better and better and better,
a lot of the techniques that we were using
in the film space started to translate very well
into the game space.
So that's where I just naturally gravitated.
- This hyperrealistic view of humans,
that's the same techniques that are also being used
in film and in gaming?
- Yes, the digital brain
actually activates digital muscles.
It uses the foundation of a system
called Facial Action Coding System.
A way to unitize each muscle pull in the face,
and those are called action units.
So we've in turn translated that into a digital space
and all of our animation systems
are based on the idea of these individual action units.
- How many of these action units
do you have in your face?
- There's probably 50 muscles in your face.
This is what we get from a scan,
which is literally millions and millions of polygons.
And a polygon is one of these little triangles on here.
But this is just way too dense.
So we have to go through and really reduce down
the amount of data that we have here.
- Basically into your 50 muscle groups
- Into the 50 muscle muscle groups.
This is what we end up using, actually,
from an animation perspective.
You can see the number of polygons
have significantly been reduced,
but also what we have over here
is we have every single one of these muscle units.
So we can just go in here and we can slide any one of these
and start building an expression.
This is what we call our DNA Studio Blender.
Now that we have all of these different phenotypes
contributing to what a digital human
could potentially look like, customers can go in here
and choose what they want their digital person to look like.
- This is very fine-grained.
So all of that is autonomously controlled?
This digital person
already has all of those controls in there.
- That's correct.
- For your digital brain to make it animate.
- Animation's really costly and expensive to do.
And so that's really where the idea
of the digital brain comes in,
it makes it possible to animate these digital characters
very quickly, easily, just automatically.
- So that's your DNA Studio.
The individuals are being created, digitally.
- Yeah.
- But then you have the operation of it
is something very different,
which is much more of a platform.
- And that's what we would call Human OS.
That's where the brain is.
We're actually seeing into the user's world
and understanding their emotions,
their body language so that it can respond appropriately.
The voice comes in, the analysis is done,
the natural language processing is done
where the speech is converted to text,
and that then comes all into the brain
at the core of this.
Human OS takes all the pieces
in the cloud, actually operates there.
And there's another piece that then comes back out,
text to speech.
Personality is applied on top of that.
We have behaviors that reflect different types
of characters and then it goes to the next step,
the visual aspect.
The output is a motor control system,
your facial muscles and everything else.
You could have a totally scripted conversation
but they also could be computational subroutines.
They could be doing math, or they could be a search engine
because the digital person's goal here
might be to be a greeter on your website,
and then it would help you navigate.
- So from that point, you go into animation.
You say you call it autonomous,
and so latency is very important.
- Latency is always a problem
with something like this.
You want to get it as close to real time
as you possibly can.
With cloud technology,
the network latency isn't the biggest contributor.
It's the speech to text is probably
the single most challenging piece to it.
- Okay.
Until now, all our interfaces have been driven
by the capabilities of the machinery.
Screen, touch, keyboard.
Because the computer forced you to.
But being able to go back to a more natural interface,
combining biology and cognitive sensors
with digital technology is, in my eyes, very exciting,
and can unlock digital systems for everyone in the world.
- It's been such an honor
to meet an architect of the cloud.
I can't wait to see what you do next.
- I'm interested to know what you will do next, as well.
- Thanks, Werner.
- So, all very interesting.
The roots are in CGI world.
What is the kind of real use cases
that you were thinking about?
Like, oh, these customers really need this product.
- I'll give you another example.
We're working with one of the biggest
food ingredient companies in the world,
and they don't really have
direct relationships with their consumers.
We created a digital cookie coach.
And so you can actually go
bake chocolate chippy cookies.
- Together.
- We can build the same sort of emotional connection
between digital people and real people
to enable these sort of really,
really high value types of relationships.
We're doing something similar in the entertainment space
with our celebrities.
So, you know, we all have our heroes.
Most of us won't get the opportunity
to meet that person in real life.
But imagine a world now where we can create,
and this is literally what we're doing,
We can create a digital twin of a celebrity.
We've created a digital twin of Jack Nicklaus.
Jack's digital twin is 38 years old
as he was at the prime of his career.
You know, within 12 months, you'll be able to go step
onto the first tee box at Pebble Beach.
Digital Jack will be standing there
and he'll be able to tell you how to play the hole.
Take another, you know, 12, 20 months,
and maybe you're playing a round of digital golf
with digital Jack Nicklaus at digital Augusta.
You're having a completely immersive experience,
and what's important about that is that's an experience
that almost nobody could get to have in the real world.
- How do you see this progress over time,
not necessarily as your product,
but sort of that whole space?
- In terms of changing the paradigm
and the way in which we create CGI characters,
the way in which we animate them,
that's our field of expertise.
And then, you know, in order to bring
these animated characters to life,
we use a whole range of different AI technologies.
AI-based voices or synthetic voices.
That's another area of artificial intelligence,
or natural language processing engines.
- Fortunately these days,
we provide these as products, instead of that
you have to figure that all out yourself.
- Or build it all yourself, yeah.
- Or build all yourself.
- AWS is really, really important to us
in the way in which we are able to render our technology
in the cloud and deliver it to literally millions of devices
around the world.
- These technologies that we only used to see
in science fiction movies
are no longer fiction at all.
As we spend more and more of our lives
interacting with machines,
AWS technologies are at the center
of making that relationship more natural,
realistic and empathetic.
From retail and entertainment to healthcare and education,
Soul Machines' digital people offer a boundless vision
of how we might engage with machines in the future.