when i first started working on the
problem of trying to create a photoreal
digital human it's in the late 1990s and
we actually didn't know if it was ever
going to be possible it's still possible
at that point to imagine that there was
something
magical or ethereal or even spiritual
about just the way that a human face
looks that somehow technology would
never be able to simulate
now we can do something that we didn't
even know was going to be possible
here at the institute for creative
technologies our focus is really on the
next generation of virtual reality we
think of our mission in some ways to try
to create new technologies that'll get
us further toward the star trek holodeck
which is this room that you walk into
and it's the perfect ultimate virtual
environment where the walls fade away
and suddenly you're somewhere else
talking to digital characters
interactively
we keep reaching limits with digital
culture and thinking um and thinking
it's over we've gone as far as we can
virtual reality and now digital humans
creating these visual experiences really
does show some promise as like a whole
brave new world for inspiring creativity
and new dimensions of digital experience
right now i'm inside light stage x it's
a computer-controlled lighting system
that we use to digitize human faces at
the highest resolution possible
it's like you're getting an orgone box
or something like that and being
bombarded with cosmic rays they're able
to capture all your expressions 360
degrees all the follicles of your hair
and everything and so it can basically
duplicate you
so human faces are really difficult
first of all because we have so much
experience looking at human faces in our
daily lives there's special mechanisms
in the brain just designed to recognize
people and to try to pick up very subtle
details of how somebody is moving their
face as a cue to what they're thinking
and all the forms of non-verbal
communication that's there so if you get
anything off you could very easily
pick it up as a cue
that something's wrong inside this
person that you're looking at
one way to think about this is the the
famous concept of the uncanny valley
which was
originally applied to robots that are
trying to look more and more like humans
something that looks a lot like a human
but it's got something wrong with it
that maybe you just can't quite put your
finger on can actually be a lot creepier
than a cartoon character that isn't even
trying to be fully human one famous
example
was the christmas movie polar express
the motion of the faces and how they
mapped the low resolution motion capture
data under these digital characters
general consensus was that the
characters look a little bit dead or a
little bit creepy and that's not good
for a christmas film
the company image metrics approached us
in 2008 about hey do you want to try to
create one of the first photo real
digital actors that became the digital
emily project we suggested well how
about we scan the actor not just in a
neutral pose let's actually
scan the face
in every facial expression that we could
imagine for this actor they cast emily
o'brien who at the time was on a soap
opera called young and the restless and
in march of 2008 we scanned her in about
30 different facial expressions we
processed all of this data down to the
level of skin pores fine creases
4k texture maps and delivered it to
image metrics where they went and rigged
all of these different facial shapes
into an animated character and the
result is at least convincing enough
that
when we showed the final thing to emily
she just said
that's great that's just video of me
right
well i think that they have a long way
to go there's always kind of an arms
race where technology goes a step beyond
seemingly our capacity
but then we also learn to outsmart it
and the same thing has to happen when
you're confronted with a digital human
at first
you feel this
like kind of terrified sense of
submission to the spectacle you know
like
oh now i have to accept this thing as
human and it's triggering all my
emotions and i feel like a sense of
vulnerability then i begin to know i
learn
through my non-artificial intelligence
to keep up with it
so we've actually come a long way since
digital emily and if you compare digital
ira to digital emily we're now doing the
whole head and
we can render this in real time
every single frame of the emily
animation took over a half hour to
render with digital ira this runs in a
game engine and it can actually generate
these at 30 frames per second so going
from 30 minutes per render to 30
renderings per second is orders of
magnitude speed up
certainly the most important part of a
digital character is the eyes and by
this i don't mean just physically the
eyeball itself but the whole area of the
skin around the eyes there's all these
little muscles here that can kind of
twitch and squish there's so much
information just in this little area
that is the most closely
associated with what's going on in the
brain
the computer graphics rendering
techniques for digital ira's eyes are
incredibly advanced the translucency of
the skin around the eye the fact that
the front surface the cornea of the eye
is smooth whereas the sclera the white
of the eye is a little bit bumpy
all of those things are being simulated
as faithfully as possible
in the world of acting if you're acting
on the stage you're acting with your
body if you're on television then you're
acting with your face and if you're in
movies then you're acting with your eyes
that's where so much of the information
is conveyed i ordered a yoga parfait and
the whole parfait was fruit frozen fruit
there is no yogurt it's supposed to be
half fruit and half yogurt the whole
thing is frozen fruit it could be a
little bit strange because they would do
the capture and then i would walk by the
offices of of the team and they say hey
you should see what we're doing with
digital art and at some point you know
it felt a little bit too personal where
they're almost have a voodoo doll that
they're sort of manipulating and doing
things without my knowledge so i
suggested to paul that he reverse the
letters my name and call digital ira and
create some kind of distance between
me and my doppelganger
a couple years ago i went to sundance to
see
um a piece of interactive journalism
called hunger in los angeles no you're
blocking the
you should be in line where your number
is
you put on the headset and instantly you
were walking among
people in a kind of soup line and one of
them i think went into a diabetic coma
it was a beautiful and incredibly
engaging spectacle
first
i was hit with nausea and that sense of
dislocation when i put it on
i wanted to send a signal to myself that
i wasn't buying it
and i thought that might be a way to get
my equilibrium back so one of the things
i wanted to do was do things that i
can't do in normal space you go up to
someone that's not real there are no
consequences you just want to like do
that you can't do to people i sort
of put my hand through the hologram and
then just pushed this guy i remember
you know nothing happened
but i had a sense of just like
see what's
like not real about this world
the idea for the ict came about from the
u.s army being interested in essentially
a flight simulator for social skills the
ability to interact with the virtual
character in a way that will train you
to interact with real people they had
the idea that there's a lot of
commonality for what is necessary in the
next generation of video games and the
next generation of hollywood visual
effects
probably the biggest movie that we
worked on with the light stage
technology was james cameron's avatar
and it's the first movie that we used
our newer high resolution facial
scanning process our friends at weda
digital heard about this and in 2006 we
had
sam worthington zoe saldana coming over
and getting their faces scanned
so that there would be high resolution
data available of them for the weta
digital team to build the navi
characters of course avatar there's a
tremendous amount of motion capture in
the film for the climactic battle when
i'm in my armor mobile platform my amp
suit that's all done through motion
capture so in fact the kernel quartz you
see
in the suit is me but it's an animated
version of work that i did on a motion
capture stage everything on pandora is
digital though the jungle the
uh the amp suit the lighting
it's a lot easier if the character in
there is also a digital asset because
then you have complete control over
putting them in there
i complimented myself and how good i am
in
those scenes which is to say i'm
complimenting the animators who captured
me and applied me that brought me to
life you know a lot of actors are
wondering wow is this technology going
to replace actors
every digital character you've seen in
in a movie so far or a video game if
you are believing the performance and
reading some real emotion out of it it's
because there was a real actor that
actually gave that performance in a
performance capture system i think that
we're a very resilient
uh profession
and we've dealt with all kinds of
technological change from the very
beginnings
and we've always
managed to incorporate it and use it to
our advantage much of what we do is
really about pretend
i mean it always has been and there's no
better kind of
venue than the volume which is where the
motion capture happens you get right
back to the fundamentals of acting we
don't yet have good enough artificial
intelligence algorithms to generate
all of the things that a digital actor
should do with their face in order to be
fully believable
some of the work that's going on at the
university of auckland with one of our
former colleagues dr mark sagar he's
trying to create a digital baby he calls
baby x it's based on his own young
daughter
that is going to model what's going on
in this simulation of a human brain and
eventually is going to start to be able
to speak and put words together it's a
kind of a radically different approach
to the artificial intelligence problem
what's this
good girl and sifting is what the word
is i think that's going to be some of
the most interesting research that'll
happen over the next decade or two
with this kind of 3d data you actually
have the opportunity to manipulate it
and make the head do things or say
things or do something that wasn't
actually done so it's a little bit more
important that you have control over it
than if it was just a film
it certainly will be possible
in the future to you know pick some kind
of political figure get a good voice
impersonator tell them to read any line
of dialogue you want and uh you know
create video of them saying something
that they never said
well we're going to have to become
comfortable with the fact that that
theoretically could have been created
with a computer i mean i've joked about
it
before saying well it'd be strange if
like 30 years after i'm dead i show up
and you know doing porno or something
you know it's like any technology it'll
have its interesting uses it'll have its
terrible uses and and somewhere in
between
to me it's exciting at this this period
in time right now we're really moving
ahead at a very fast pace and i really
think in the near future we're going to
see these photo realistic people
another technology working on developing
is for holographic like display of human
characters and we have a system that
allows you to render a real person that
was filmed on video or a digital
character
in a way that kind of feels like they're
really you know in the room with you
it's going to be very exciting when we
can create interactive versions of these
characters
i can imagine having
you know an incredible experience one of
these you know resurrected digital
humans of you know someone who's died
being able to talk to
you know my late grandmother not because
it would feel like it would trick me
into thinking i was actually talking to
her but because i
would love to see her again and to the
extent that the creators of digital
humans
can recognize that this is this is an
aesthetic experience these things will
be much more successful than we ever
anticipated and it might very well
change our understanding of who we are
and what it does mean to be a person
maybe it really is that you are a set of
atoms that exists on this earth with an
ability to produce and give and
contribute in some way
and that that's going to distinguish us
from any kind of computer algorithm
you