>>Lawrence Jones: Hey, everybody.
Welcome to Using UE4 to Plan and Execute Choreographed
Performances. My name is Lawrence Jones,
I am the Executive Creative Director
of a company called Sequin AR. A little bit about Sequin AR.
We create new formats, existing formats,
and live event experiences for broadcast television.
By new formats, I mean,
we partner with production companies, talent, and networks
to conceive, develop, design, and execute live AR shows.
With the help of evolving tech like UE4, limitations
from the past have now gone away for making this possible,
and now making revenue opportunities in Hollywood.
Existing formats, we help producers and production
companies enhance their existing formats
by using AR to tell new stories.
Shows are trying to stay relevant,
and we are seeing trending elements in this space
such as AR-immersive explainers, AR set extensions,
and AR war tables to name a few. And live event experiences.
We work with musicians, sporting events, promoters,
and e-sports companies to make their live events pop in AR.
Our job is to help these performers go viral
and keep the conversation going.
So, here are some examples of high-profile AR projects
the members of Sequin AR
were heavily involved in at previous companies.
You might recognize the Weather Channel's Storm Surge,
had a billion views worldwide, and took home a gold BDA award.
Super Bowl AR Open this year was nominated for a Sports Emmy
and now a Broadcast Tech Award.
We worked with Injustice and Street Fighter, Returner,
League of Legends World's Opening Ceremony,
you might have seen that.
And finally, the Weather Channel's Tornado,
which last week just got nominated for an Emmy as well.
So, we are really proud of the previous work.
So now, one of the first projects we took on at Sequin
was for an artist named Madonna. You might have heard of her.
We are proud to have heard this week that it, too,
was nominated for a Broadcast Tech Award.
So, going back a little bit, I want to discuss a couple--
one key thought that we all have here at Sequin, is that,
you know, there are two buckets of AR in our world, you know?
Two experiences, two major buckets.
One is linear story driven by narrative,
usually by music or script, and there are interactive ones,
driven by, say, a live read from a talent
that has unknown pacing and needs the ability
to trigger sections of AR when needed.
An example of that interactive kind
would be the Storm Surge we discussed a minute ago.
Other examples of linear would be Super Bowl Open,
the League of Legends Open, and now this Madonna project,
which was completely driven by music.
So, the time code of the music bed
drives everything in the performance
including monitor content, projectors,
practical lights, et cetera.
So, it is imperative that it drives our AR renderers as well.
So, let us talk about the creative brief.
Five Madonnas on stage at once.
We were asked to put five Madonnas on stage at once,
including four that were digital personas of herself,
the bride, the playful, musician, and spy.
These will be choreographed with multiple performers.
Super tight choreography had to be designed
as not to give away any tech limitations of the AR.
Madonna had the best choreographers
in the business working on this,
and it was great seeing them so passionate about this project.
The personas need to interact with stage pieces.
There were shiny floors both center stage and stage left,
small cocktail tables,
and a satellite stage that was small
and had its own set of limitations
so that we could not have the reflections clipped
by the small size of that stage.
We had three AR-enabled cameras. In our world, any live camera
that needs to have AR incorporated
needs what is called telemetry.
For this piece, we enabled three cameras,
the front of the house,
the wireless Steadicam, the jib crane.
And, Justin will go more into this in a minute.
Multiple environmental set extensions
were required by Jamie King,
the creative director of Madonna,
and he wanted to extract the places
represented in the song Medellin,
and incorporate these environments
in the practical stage set.
Festooned lights, clouds, foliage,
set extensions are all examples of what was created.
Jamie also asked for fireworks, rain, butterflies, and clouds
to bring more organic beauty and life to the piece.
And finally, dynamic transitions were incorporated
as a way to reveal and hide these holograms,
as the media coined them.
Niagara Particle Effects was used heavily
for things such as gold dust and blurry effects.
So, as any groundbreaking television project,
it takes many teams and people to successfully accomplish it.
Dick Clark Productions was our client along with Madonna,
and we worked side by side with Dick Clark
on integrating technically
and operationally with their BBMA show.
They were incredibly hospitable
and open to breaking new ground with us.
Dimension, their volumetric capture studio in London
was where the personas were acquired.
We will discuss that more in a bit too.
Arcturus, these guys were in LA
and were tasked with VMC cleanup, task automation,
and play out optimization.
I will also go deeper into that in a second.
Reflector, this studio is in Montreal,
and in charge of Asset creation,
from the set pieces to animation.
Brainstorm 3D.
Justin will go more into detail with them,
but they are a real-time broadcast graphics
software from Spain, and we chose them
to be the UE4 broadcast layer for this project.
And finally Stype, they handled the camera telemetry.
It is a complex field, and these guys from Croatia
made sure that the tracking was locked on
from all cameras simultaneous. So, main challenge number one.
This AR experience needed to be
so perfectly executed from multiple cameras
at specific times and exactly in the right real-world spaces,
and live performers and dancers needed to interact with it.
So, planning was essential.
Challenge two, it was specifically
that we had the choreography already planned
and given to us by Madonna's creative team
when we started this project.
So, we needed to help them figure out how to design shots
that can physically work to be impactful.
Here is a clip. It is a little stuttery, sorry.
It is a clip of the original rehearsal
that was given to us where the dancers of Madonna
were used as place-holders for the stand-ins--
I am sorry, stand-ins for the actual personas.
So, the solution for this,
much like all of us here, is the Unreal Engine.
We created a UE4 planning pipeline.
So, the amazing thing about working with UE4
as a Realtime renderer in this world
is that there are tremendous efficiencies for us
since we can create fully CG previs environments
that can then over time evolve as the final product
without moving to another software.
Essentially in the end, removing the CG stand-ins,
and replacing them with the live
practical elements version of themselves.
For example, we can then hide the CG stage and replace it
with the real stage through the eyes of the camera.
So, here, you will actually see a techvis
that we created for the director to see,
in order to get really cool shots at the lensing
that they had provided with us,
that they are going to provide for the broadcast.
This shows how far we can go and how wide
without the Steadicam guy falling off the stage.
So, unlike film, where you have previs to envision the creative,
techvis to plan out how you will execute,
due to the short timelines in our world,
we create the techvis,
and use that to come up with the creative
within the boundaries of what is possible.
So, now that we have a good idea
of what the personas are doing and potential shot design,
we are then ready to acquire the Madonna digital personas.
Volumetric capture, also referred to as 3D video,
can be 3D point cloud data, or in this case,
a 3D sequence of Meshes.
This video that contains temporal and spatial data
is derived from many video cameras shooting synchronously,
and after cleanup and some special attention,
these Assets are created.
As mentioned, Dimension Studios in London
was the capture company.
The teams went there and shot many different
takes with Madonna.
Decision lists were then created by Jamie and Carla,
the creative directors of Madonna,
based off the 2D video,
where then the data from all cameras
were then processed by Dimension
and sent to Arcturus for data cleaning,
fixing, and optimization.
So, those Assets were then sent to us,
where we would then replace our persona stand-ins
in our previs with VMCs as well as the performers
with live-action reference material from second rehearsal.
The footage was keyed, stabilized, and placed on cards.
We would hold many sessions with the client
to tweak the choreography and shot design
with the help of UE4's Sequencer.
So, Sequencer was imperative for this project.
I spent much time in there, and without it,
we would not have been able to finish this project.
So, while this process was happening,
teams such as Reflector would be iterating
the other AR elements based on the client's feedback,
continually working on the models, Textures,
transitions, particle systems, and animations.
This was happening all the way days before the performance.
And as you notice, we call this section livevis.
We do not have previs, we do not have postvis.
We go directly to techvis,
and at a certain point do a hand-off to livevis,
because this is what will be seen on TV,
with lots of iterations up to that point.
And here you saw, this was our first techvis
that we swapped out the VMCs for the stand-ins.
Here, in this image,
you can see where we replaced the static mesh of Meluma
with a keyed card from his live stand-in at second rehearsals.
So, without further ado, up next we would like to talk,
we would like to bring up Justin LaBroad,
our VP of engineering,
who will go into a deeper tech dive for all of you all.
[APPLAUSE]
>>Justin LaBroad: Hello, everyone. So, on-site planning.
In addition to the previs, much of the work
is pre-planning execution at the event itself.
Pre-production is half the battle for us.
We worked closely with Dick Clark Productions
in setting up times where we can rehearse
and planning technical setup around them.
We have a very condensed timeline
and limited opportunities for technical testing and execution
before we have to go live.
We also do not really have the luxury of being able
to re-shoot anything a day late or even a few hours late.
In the world of live broadcast,
we do not get in the way and impact production minimally.
There were several other performers' rehearsals,
aside from Madonna, we had to account for,
and only were able to get the full staff
at a few time periods before going live.
Each rehearsal we had to treat as if it were live
because of the limited time.
The broadcast has many moving parts.
It can be a challenging environment.
For example, imagining as an Unreal Engine technical artist,
you have to do your normal day job,
Blueprinting, creating content, however,
you are crammed into a small production truck
with tons of people on top of you all day.
So, for camera telemetry,
our three live augmented reality cameras
require camera tracking technology.
For this project, we chose Stype camera tracking.
Our three cameras were a Steadicam for up on the stage,
a jib camera as pictured here,
and a further back pedestal camera.
StypeKit was used for the jib, which has sensors installed
on all the pivot points of the jib itself,
relaying the movements of the jibs to the tracking unit,
and sending out the final telemetry
to the Unreal render engine.
Our pedestal camera uses a hardware head,
which would track the pan, tilt, and zoom of the camera.
The telemetry systems themselves would handle
the unification of the individual coordinate,
relative coordinate systems of each of the cameras,
and relayed it into one unified coordinate system
inside Unreal Engine itself.
For our Steadicam, we used Stype RedSpy.
This is an infrared camera that tracks small reflective stickers
that can be placed on the ceiling or the floor.
Ahead of time, a map of the sticker locations is created,
which allows us to know exactly
where the camera is inside that space.
Generally, you can put the reflective stickers
on the floor or the ceiling.
In many studios, they are generally on the ceiling.
But for on-air stage performances,
we tend to prefer them
on the floor of the stage for easier setup.
And this picture is showing the Stype RedSpy unit
mounted on the camera, kind of pointed downward.
Once the previs is planned, we can then begin determining
where the reflective tracking stickers
are going to be placed on the stage.
For the Steadicam operator, we need them to be on the front
and side areas of the stage. Generally we blanket the stage
with stickers farther outside the range
to give the camera operator some flexibility in their moves.
There is usually a great variation in the surfaces,
and some hold the stickers better than others,
and even though we will try to determine ahead of time,
there are always surprises with the surfaces
we are trying to put the stickers on.
A big concern early on was the large number of performers
and rehearsals that were going to be on stage
performing after we calibrated all of our cameras.
There were a dozen or so performers rehearsing two
to three times over the course of a few days before the event.
For example, Taylor Swift had an entire band
using areas we had already calibrated,
and recalibrating or remapping the area each day
was not an option because of the tight schedule.
So, we explored several options, including a Marley floor
that could be rolled up and stored
when Madonna was rehearsing or performing.
Our concern with that is that the stickers may fall off
from the constant rolling and unrolling.
So, we set up a test area inside our studio
and started laying down stickers,
and fortunately, after testing this in our studio,
we found they were extremely resilient to foot traffic,
even after we were intentionally trying
to remove them with our feet.
Lots of help was required, but once we had the plan ready,
we were able to enlist a lot of help from Dick Clark Productions
and put down about 2,000 stickers in less than an hour.
Silverdraft has been a great partner for our PCs
we use at both our studio and on-site events.
We generally try to go with the latest processor
and NVIDIA cards at the time.
For this particular project, a fast SSD was important,
because we were trying to stream
four volumetric capture videos at the same time,
and caching the entire performance ahead of time in RAM
was not an option. Unreal Engine is not fully ready
for live AR broadcast out of the box yet.
In the past year,
the base functionality required for live broadcast
has been slowly been implemented into Unreal Engine,
and we expect a lot more steady progress on that
in the next year.
So, some of the things we need for Unreal Engine
for live broadcast are HD video input-output,
the ability to composite the AR with the clean camera,
shadow reflection catchers on visible geometry in the scene,
and control of the application,
control of all the renderers from an interface
that we can use in the production control room.
So, for this project, we chose Brainstorm
for what we call the broadcast layer for Unreal.
Brainstorm has been around for a long time
in the Realtime virtual set space,
and currently has an Unreal Engine integration
with their product Infinity Set.
We chose Brainstorm over some of their competitors
because they had already integrated
external control of timecode
into their Infinity Set software,
which we really needed for this project,
and had their software running on UE4 4.22,
which was the latest at the time.
They have a Python-based user interface,
which is very flexible
and allows you to build and create,
even do programming on the fly,
fast user interfaces that we can use on site,
and having an external application for saving
and loading Unreal Engine Actor presets
such as transference properties, and can call Blueprint functions
is extremely important for a live broadcast world.
We cannot really spend a lot of time
bouncing back and forth between the Unreal Engine Editor
and running Unreal Engine in Game Mode,
which we run for performance reasons.
There are many on site creative changes happening
when we need to make adjustments and save those,
and it is a great time saver
to manage that in an external application.
The ability to do all this while controlling
and connecting the multiple renders at once is also a must.
Based on everything I just talked about,
this is a little bit of a system overview
of how we actually plug into the broadcast on site.
We used just one Unreal Engine 4
and camera tracking system per camera,
so all viewpoints are available to the director at all times.
Camera tracking data gets distributed over the network
at 59.94 frames a second.
We then composite in the Unreal Engine the Unreal Engine
output with the camera video inside Infinity Set,
and the resulting output comes out our Aja Kono 4 cards
into the video switcher for the broadcast for the directors.
Our Infinity Set control PC is separate from our render engine,
and serves as the orchestrator for any events and actions
that need to occur during play out.
Our timecode comes from a Master ProTools audio track.
This is the same audio track that feeds timecode
to other lights and monitors inside the arena.
So, our sequences are set to go based on that.
When we are ready to go live,
we are actually hands off everything,
and let the production trigger the timecode,
which is going to trigger all the sequences.
At Sequin, we are currently developing tools
to make our live productions easier
and require less development.
Our first application is called simply Timecode,
and it is an external application
with a companion UE4 Plugin
that consumes timecode over a computer's audio port
or other LTC devices, and allows you to set up
triggers and events and control Unreal Engine
for live broadcast AR performances.
Producer tools is another area we are working in.
There is a huge need for on-site producers
to see the final output anywhere in the event,
and make, suggest changes and placements
for virtual objects and AR characters.
We then need the ability to take these easily
and quickly back into our actual live production
Unreal Engine sequence.
So, imagine a situation where Madonna is actually looking
at Unreal Engine on a tablet,
making her quick changes for things,
where she wants things to be,
and we can actually take those in quickly.
Thanks for attending.
Please come and talk to us after the presentation,
and follow Sequin on Twitter and LinkedIn
to see projects we are working on next.
Thank you. [APPLAUSE]