Hi everyone. My name is Christian Andrade, Unreal Engine
Evangelist for Latin America.
In December of last year, a project
called 'Meerkat Demo' was published on The Marketplace
for free.
This short film, rendered entirely in Unreal Engine,
was made by Weta Digital, built to explore the highest
level of quality possible while maintaining as fast a rendering
speed in real time.
Now, I want to take advantage of the fact
that we can access this project,
and try controlling it using another product that we
have published: the 'Live Link Face' app.
The latest models of Apple's iPhones and iPads
offer facial recognition and motion tracking
capabilities.
It captures the position, topology,
and movements of over 50 specific facial muscles.
If your iPhone contains the 'TrueDepth' camera
and is compatible with ARKit, you
can use the 'Live Link Face' to control
complex facial animations on 3D characters inside Unreal
Engine.
Here, I have previously downloaded the 'Meerkat Demo'
and I have already opened it in the editor.
The first thing we must do is [INAUDIBLE] plugins.
We need to activate Apple ARkit, and Apple ARkitFaceSupport,
LiveLink, And finally, LiveLinkCurveDebugUI
I should point out that this project was not
created to be animated using facial motion capture
systems, but as a short film using the Sequencer.
Therefore, we will have to improvise some things in order
to make it work properly: for example,
this project does not have an animation blueprint.
So we have to create one that controls the animation
of the meerkat mesh.
For this, we just need to right-click the Skeletal Mesh
in our Content Browser and choose the option Create, Anim Blueprint.
Now, let's go to the 'Meerkat Actor Blueprint' and replace
the 'Animation Mode' and 'Anim Class' associated with
the skeletal mesh.
In 'Animation Mode', let's choose
'UseAnimationBlueprint' and for 'Anim Class,' look
for the 'Animation Blueprint' that we just created.
Once done, let's go back to the 'Map Editor' for a moment.
In the 'MenuBar' we will click on 'Windows' and choose
'Live Link.'
This opens the 'Live Link Connection Window.'
I'm going to drag and drop it over here.
If your app is properly configured,
and your iPhone and computer connected to the same IP
address, you should be able to see
that Unreal is receiving data from a device.
Otherwise, it would be blank.
The green dot indicates that the frame transmission
is consistent and successful.
Now let's open 'Anim Blueprint.'
Now we can add the 'LiveLinkPose' node
to the AnimGraph.
This node is used for transmitting the animation
data to the final pose.
In the drop-down menu, select the 'LiveLink' connection
that comes from your phone.
If it appears empty, it is because the Live Link is not
streaming yet.
Here we have our first troubleshoot--
This mesh was not intended to be used with a facial mocap.
ARkit requires all the names of the BlendShapes
to match with the 51 names that the Apple Arkit SDK detects.
Fortunately, Unreal allows us to create 'Remap Assets' to solve
this.
In order to create a 'Remap Asset'
we must first go to the 'Content Browser'
and right-click to open the context menu.
We then press the options, 'Blueprint Class' 'All classes'
and create a new 'LiveLinkRemapAsset.'
Open this asset, and we are going
to overwrite the 'GetRemappedCurveName'
function.
This output pin returns the name of the 'Blend Shapes' curves
according to ARkit's configuration.
And with this input pin we are telling
Unreal to override with this name of the mesh.
Now, how are we going to remap several curves?
The most efficient way is to use a 'Switch' node.
If we want to know the names of the curves of our meerkat mesh,
we only need to open the skeletal mesh
and explore each blend shape, one by one.
Here we have a new problem:
this mesh was not made to be controlled with face mocap.
So it is far from ARkit's recommendations.
This type of problem can also be common
when working with other Marketplace character assets.
It is possible that some assets even
have a very small number of morph targets.
Some may not even have muscle movements assigned
as blend shapes, but to expressions such as
joy, sadness, or anger to name a few.
In any case, this problem can also
be addressed in a creative way.
It is just a matter of identifying which expressions
on the meerkat most closely resemble those of ARkit's.
And this is where the 'LiveLinkCurveDebugUI' plugin
comes-in handy.
This plugin gives us access to a node called 'Display Live Link
Debugger.'
We are only going to connect it to the 'Begin Play' node
of my 'Animation Blueprint' and we are then going to press
Play.'
This will automatically open a tab
where we can see the complete list of the 51 curves
that are transmitted via 'Live Link' from your device.
And this also shows us the value of each
of these curves in real-time.
I'm going to drag and drop this window here
into the skeletal mesh editor of the meerkat.
This will help us compare which bar
is the one that grows the most, when we try to imitate one
of the meerkat's expressions.
Let's see an example.
Let's focus on the nose.
If we try to imitate the movement it makes when
modifying the value between "0" to "1"
of the 'Nose Wrinkler' curve,
we can see that among the curves
associated with the nose that react the most are
'NoseSneerLeft' and 'NoseSneerRight.'
This happens because ARkit detects them separately,
both on the left and right side of the face.
The curves of the meerkat are more limited.
Only a single curve is assigned for both sides of the nose.
So adding either of the two ARkit curves to the remap asset
is enough.
In this case, I will add 'NoseSneerRight'
and assign it to the 'NoseWrinkler' curve.
Be very careful to write them correctly.
For the sake of this video, I have already taken 14 curves
and added them to the remap asset.
Feel free to try using other curves if you like.
Or try to connect some of these to curves like the [INAUDIBLE].
I will stop for a moment on this screen
so you can pause and copy it
Are you ready?
Let's continue.
Now that we have the 'Remap Asset' ready,
we only need to add it to our 'Live Link Pose' node.
In the 'Details' panel, we simply click on the node
and replace the retarget asset with the one we just created.
We compile, and we are now ready to test it.
Let's verify again that the panel of my Live Link connection
has the green light - which indicates that it
is receiving data correctly.
And since I no longer need the debug UI plugin,
I am going to delete the display Live Link debugger node.
And voilà, its alive!!!
And looks great.
At this point we have control over the blend shapes
of the meerkat, and we have connected some of these
to the curves of the native poses
that we transmit from the Apple ARkit SDK.
However, we cannot yet move other elements that depend
on the rotation of the bones of the skeletal mesh.
In this case, the neck, eyeballs and teeth.
If you check the documentation for 'Live Link
Face' or the 'Face AR Sample,' you will find that we recommend
using a 'Pose Asset' to enable transferring the information
from these blends between the values "0" and "1"
to the bones.
In this particular case, the Meerkat demo
does not include an animation with the movements of the face,
or a pose asset that allows us to extract these movements
and assign them a curve that allows us to control them.
To solve this, we will have to be more creative.
The 'Transform Modify Bone' node can
be used to modify the 'Transform' property
from a specified bone.
Let's add one in our 'AnimGraph' and assign the 'Head'
as the bone to modify.
This will allow us to control and modify
the location, rotation and scale of this bone.
We don't need location and scale so I'll uncheck this
on the 'expose as pin' option .
It is important to verify that these pins are ignored too.
And the rotation must be set as "Add to Existing"
Now, in order to get the additional Apple ARkit blend
values in our blueprint, we can use a node called 'Get Curve
Value.'
We are going to add three of these in the 'AnimGraph.'
The names of the curves that detect head rotations in ARkit
are 'HeadYaw,' 'HeadRoll' and 'HeadPitch.'
We will need to offset the value of these curves a bit
so that they can rotate the correct value of degrees
to the head bone.
The values I have here were prepared in advance
but feel free to try other values you prefer.
The rotation of the head model also
needs to be somewhat adjusted.
So we will connect the 'HeadYaw' to pin
'Z,' the 'HeadRoll' to pin 'Y' and 'HeadPitch' to pin 'X.'
But first, we still need one more detail.
As mentioned before, if the meerkat does not
have a blend shape with the same name as that of ARkit's blends,
it will not be able to transmit the information.
So if we try to use it at this point, it will not work.
To solve this we only have to add these three curves
to the 'Anim' blueprint.
In the menu bar, select 'Windows' and the 'Anim Curves'
tab.
We right-click, add a new curve and give it the same name:
'HeadYaw'.
We repeat with the other two curves.
Once done, we compile and we should
be able to move our head.
If it doesn't move, go to the 'Live Link Face'
app on your iPhone and check if the option 'Stream Head
Rotation' is disabled.
We can use this same method for the eyes and teeth.
In this case, we can rotate the bone of each eye,
or you can create a material that moves
the iris of the eye in specific directions.
We could work on this now, but it
would overextend the scheduled time for this video
so we will not review it.
But we do have a Meerkat that can move it's head
and does facial expressions as well, and looks great!
Now, let's move on to our last step.
The meerkat has four blend shapes
that allow it to simulate the movement of breathing.
Let's do something to enable animation on this.
The 'ModifyCurve' node allows us to modify
the value of a blend shape of one's controlled mesh.
Let's add to this node the four curves that simulate
the "breathing" of the meerkat: 'BreathingChest,'
'BreathingNeckHigh,' 'BreathingNeckLow' and
'BreathingStomach.'
ARkit does not have a sensor that
detects when one's breathing.
But we can do something that can simulate it.
In order to modify the values of these curves,
we will add a 'GetRealTimeSeconds' node,
And from here we can obtain the Sine of the time in seconds.
*Hmm...
I'm going to exaggerate this by multiplying the value by three.
By the way, feel free to try other ways of doing this.
Maybe a 'MapRangeClamped' or other methods could work.
In this case, I think that with this method
I will get something functional and quick to show.
And with this, the breathing simulation
should be convincing enough.
'Live Link' and 'Live Link Face' app
are free tools that allow us to democratize motion capture
and animation in Unreal Engine.
With this, I hope you can create amazing and funny meerkat
videos.
Molly the meerkat, signing ouuuuuuut!