>>Chad Reddick: Hello SIGGRAPH 2019!
We are Digital Domain,
and we would love to talk to you about Building
a Real-Time Digital Human Framework for Live Events.
First off, new technology requires frameworks.
On the left here, you can definitely see
our Autonomous Camera Framework we used at TED 2019.
On the right here, we used our end Character system
where we can have more than one Character in the scene
that we used at SIGGRAPH 2019 Real-Time Live!
First off, before I go into more details,
I would like to talk to you about Digital Domain.
We are a visual effects company
that worked on film and commercials
and also a digital human real-time.
Digital Domain has worked on films such as Avengers:
Infinity War and as well as Ready Player One.
Who are we? My name is Chad Reddick.
>> I am Jason.
>>Chad: And we are Senior Unreal Engineers at Digital Domain
at the Digital Human Group.
>>Jason: About Digital Human Group.
What we do is we work on trying to make the most realistic,
digital humans that we can possibly imagine.
This involves the best skin that we can come up with,
the best facial movements, and hair.
But what contains a digital human?
>>Chad: A high-quality 3D model,
the movement of the digital human,
and of course, the facial movement of the digital human.
>>Jason: But what about the details?
>>Chad: The details, we need to have the blood flow,
what gives you that real-looking kind of blood movement
in your face.
You need good lighting on the digital human,
the realistic look on the specularity,
and everything else, and of course shader complexity,
which there are much,
a nested things that have to go into a digital human
to make it look realistic.
Of course, here is the result. We have Doug here, DigiDoug.
He is smiling, waving, saying hi to you guys.
>>Jason: At this point, our framework
is just one Character at a time.
It is non-networked, and it is either flat or VR only.
>>Chad: Let's talk about TED’s live event.
One day, Doug comes into the loft.
He says, “We got invited to TED!”
And we were super excited to hear about that.
This would be the first live drive of DigiDoug and Elbor.
We were really excited to have this.
Then we got the features requested.
It was about three weeks for when TED was live,
about two weeks of development time.
We had to build Autonomous Cameras and User Cameras.
Autonomous Lighting to properly light DigiDoug
and Elbor in runtime. This required a new framework.
We had to create a modular way to create Cameras
with autonomous options, meaning kind of like an AI
where it creates a perfect view of the Character
but without anybody using them.
Same thing with the lighting rigs.
We had to make sure when the Character was moving around,
we had to make sure the lighting was always perfect.
We want to use an approach of using IoC,
which is inversion of control.
Then from the top level to the bottom level,
everything was initialized, everything had a context,
and everything moved on really fast.
It also needed a name,
so we called it the Director Framework.
This is a little bit complex, so we had the director,
which contains all the scenes to view
and also the master input.
Whatever the director said, if it was,
hey, I want you to change this shot here.
That is kind of the concept of using the input.
Also, it always had all of the views so that the director knew
what shots we were going to be taking.
Let's talk about the scenes.
The scenes contain the Character, Camera,
and lighting rules and references.
What does this mean? This means that we need to know
what Character we are looking at,
the Camera type we are using, if it is going to be autonomous,
going to be user-based, or is it going to be static?
Same thing with the lighting rules as well.
Is it going to be following the Character?
Is it just going to be static throughout the whole scene?
Is it going to have a light
that you are going to be able to pivot around to it?
We are going to talk about another meat and potato,
which is the CameraSceneActor.
This contains all of the autonomous user
and static options. It was able to understand
that if you wanted to view a Character,
it had to do specific kinds of ways to look at it
and give it the perfect view.
If the Character was out of view from this side,
it would go back over here. They also have user options,
like being able to move up and down,
left and right, using the mouse wheel,
using the mouse up and down,
anything else you wanted, or of course, the static option,
where it would just be a standstill Camera.
It would not move, not look or pivot,
but just be standing there.
The LightingSceneActor was autonomous or static.
So autonomous
meaning it would follow the person around like this,
be able to keep the lighting in frame,
or you have it just be throughout the whole scene
it would just stay there
and you could walk out of the lighting if you wanted to.
Here is the result.
We have the left here is Doug, and the right would be Elbor.
If you watch here, you see how the Camera tries to go over
and watch over to keep that same frame shot of you
whenever Doug or Elbor moves around.
The Cameras are able to switch at runtime.
We had a great success.
The framework added to the bigger picture.
We are able to make a framework that we could reuse at any time.
Reliable usability,
meaning that it was very structured to an extent.
We could build even further on top of it.
It was also very artistic friendly.
It did not require code to the artist
or kind of the compositor
sort of way to get a good, perfect shot.
>>Jason: At this point, we knew we had SIGGRAPH coming up.
We had the idea of, what if we had two DigiDougs,
and what if they could both see into virtual reality?
While we are at it, why not throw in ray tracing?
The issues with our current framework at this point
was hardware limitations. Ray tracing is not cheap,
and we are about to throw a second Character in there.
Our Characters have a lot of geometry.
I think we are talking about two point something million...
>>Chad: About 2.5 million, I think.
>>Jason: 2.5 million verts per Character.
So we are going to do all that, ray tracing, and at this point
we want to drive everything from the VR headset.
Next, we are not networked yet. Sorry, go back.
We are not networked yet, so we need to implement
all of our system into a replication system,
and we need to actually implement the view into VR
that is how the Characters
can see into their virtual environment.
And our previous system that Chad talked about
only considers one Character.
It does not consider multiple Characters,
so we have to now implement that into the old system.
>>Chad: This new system, called N-Character System,
we can drive multiple Characters.
We were not planning on doing 25,
but I thought, why not? Let's try it.
The features about this are namely the Mimic
and the VR View.
We will talk about those in a moment, and agnostic hosting.
This allows any Pawn, or any Player for that matter,
to be either a host or a client.
We needed a way to replicate the Skeletal transforms
because this is not natively supported in Unreal.
>>Jason: The mimic, as the name suggests,
is a Pawn that mimics other Pawns.
In this video here, on the left you can see the VR View.
That is what we are seeing from the VR headset.
On the right is the Mimic. They are pretty close.
However, there are some differences,
namely, the post processing here.
We are able to add depth of field, live color grading,
and we are smoothing out the Camera.
There are many other features we can add to that,
but this is what we used for SIGGRAPH.
Also, a really important thing
is that it saved us a ton of performance.
Instead of rendering everything from the VR headset,
we copied the transforms from the Cameras of the VR headset
over to this new Pawn.
This Pawn just rendered out from that location.
This one is my favorite, the View into VR.
This is how our actors are able
to see into their virtual environment.
They are able to make eye contact with each other,
and they are able to see
the Camera and other Objects in their world.
This changes with the size of the Characters.
They can see themselves. It is amazing.
It adds so much immersion for the conversations
that you have with the digital humans.
>>Chad: Especially eye contact is really interesting
when you are in VR watching that.
It is very uncanny.
>>Jason: All right, so one of the issues that we had
is replicating Live-Drive Skeleton.
This was not natively supported in Unreal.
We needed a way to go about this.
What we did was, we took two animations,
one on the host and one on the client.
The host side takes the data from the Xsens,
sends it into IKINEMA
to do all the IK solving.
Then this, with a little bit of extra magic,
turns into our animation of our digital human.
However, we still need to capture this final pose
so we made a custom node
that grabbed all that data at that point
and replicated it down to the clients.
The client animation is essentially
just a passive animation that waits for new data
and says, hey, do I have new bone transforms?
Once these bone transforms come in, it says, all right,
let's set this Skeleton. Now we are able to do this.
This slide might be a little confusing.
Feel free to take a picture of this
and ask us any questions afterwards.
But to quickly go after it, we have two actors on here,
Doug and Kenzie at the top. The green nodes indicate
where raw information is taken from the actors.
So the top center one, that is a motion tracking.
That is the body animations.
Then the other two, the Neural Network 1 and 2,
that is where the face data is captured.
That information is all sent down to the host computer
where the host figures out how to use this information,
sends it into Unreal,
and then sends it back out to the other Unreal clients.
Are we able to add more Dougs?
We absolutely are, but there is a cost.
More Dougs means more machines.
Every new actor will need a new suit, a PC,
and an HMC to capture their face.
All machines need to be networked together.
It does not need to be local. It can be online.
Generally speaking, what we try to do
is what we show the audience is the host.
The host has the highest quality, and everything else,
since it is not as important,
is lower quality to save performance.
Here is a short video of our real fake Dougs.
You can see them switching between Characters,
looking at each other, and they can see the Objects
that appear in their environment.
>>Chad: What is next? Host migration.
In our current framework right now,
if the host goes down or dies, the rest goes down.
We want to be able to implement a new feature
where if the host goes down,
the next client will be the next up to be the new host.
Also, physical virtual Cameras. Instead of being able to have
just using a mouse or a keyboard,
you want to be able to have a physical Camera
where you have real human motion
where we record or virtual Cameras.
Of course, more smarter AI Camera Actors.
We want to be able to have even more realistic human motion
with our Camera Actors to be able to interpolate.
It is almost like a robotic way. Make sure it feels more human.
>>Jason: Character eye improvements.
Right now, the Actors will look at a flat screen,
meaning their eyes converge on that screen.
That is not realistically how we look at things.
We need to figure out a way to be able to look at to know
what the Actor is looking at
and converge the eyes on that Object.
>>Chad: Of course, we want to record live-drive data at runtime.
What this means is we want to be able to record
any sort of motion data,
facial data at runtime with a package build.
Say we are running something down,
we have a perfect cool pose.
Hey, we want to go back and record that.
Well, they are going to be running in the XC.
I want to say thank you all for coming.
>>Jason: If you have any questions,
feel free to ask us after this presentation.
I hope you enjoy the dancing Elbors.
Thank you very much. >>Chad: Thank you.