I'm josh silverman. We capture human holograms.
And this slide show is the crazy range of holograms we've created
Over the last several years. In the next 20 minutes, i want
To cover three primary topics with you.
First, what is a hologram and how might we use it in our mr experience?
Then we will talk about the creator experience.
How do we capture and actually create one of these holograms?
Finally, the developer experience.
How do we integrate these holograms into your mixed
Reality application. So first, what is a hologram?
Well, here you see wwe superstar ram.
Can you see him from any angle. This is what it usually looks
Like when i open a hologram in our review app.
I can review the hologram from any angle.
He is actually spinning around here. He looks pretty great.
Come on. Back around. There he is.
But here is what it looks like when our partners integrate
Holograms into their experiences.
Ct-♪ ♪ ♪
♪ >> Interest is a wide variety of
Holographic content into the marketplace already that we have
Helped to produce. Some of that from the top left
And moving clockwise includes sir david
Atenborough following fossils in a british history museum.
Can you see a story-telling experience.
You can see payton manning in a gatorade transaction.
Can you see medical symptoms into a real world setting using
Hololens and students are able to diagnose the ailments.
Can you see a music video that was shot with us for a recent
Solo album. The song is coming out soon as a
Vr experience. And there is a blade runner
2049-Based vr experience which accompanied the release of the
Film p. Most importantly we have
Captured personal memories like our general manager, steve's,
Kids and put them back into an ar app on a mobile device.
I love showing people this demo in person. It is usually the aha moment
They need to really understand how the holograms can be used.
So it's time to create a hologram. What does that production time
Line actually look like? obviously we start with the
Concept. And then our stages work through
Preproduction discussions in order to turn -- we work with
Our clients to turn their concept into a hologram and plan
Out the rest of the production schedule.
We cover issues like hair, make-up and wardrobe.
Because fly-aways and wispy hair does not capture as a hologram
And does not play back very well.
Reflective make-up, wispy loose clothing doesn't work.
We work with our clients to make sure they are capturing
Something that makes sense. Then a test shoot day if
Appropriate. Finally the shoot day.
Then our technical artists associated with each stage
Willity rate through the content as it movees from captured
Content into the deliverable hologram format which gets
Delivered to our developers who do whatever post production they
Need on it and integrate into their applications.
So here is the picture of our redmond stage.
There is las stage in san francisco. That's where i'm from.
There is also partner stage. Our partners dimension studios
In london have a stage. And we are opening another stage
Soon in l.A. A stage consists of 106
Synchronized cameras that is 5 p 53ir cameras.
They typically capture 30 frames per section.
For twice the processing time and twice the storage we can
Capture at 60. We use stereo pairs to get depth
Information from our cameras using studio photo gram try.
We do not include any actual depth cameras on the stages but
We do protect infrared noise on the subject to give the ir
Cameras more data too work with. Tiago calls out rolling.
Spencer calls out action and it looks like pretty much any other
Regular video shoot. Here you see mallry hucka
Dancers. Cam are viewpoints.
Not every camera has to capture the viewpoint but that's fine.
Cameras overhead that you can see from the column but
Not underneath as it would interfere with the foot work of
The dancers.
Some people say, what happened from the bottom?
If you can't see it the cam are probably can't either unless
They are rolling around on the floor.
So this preprocessing we do before delivering, what does
That actually consist of? well we typically take the cam
Are inputs, 106 cameras.
Remove the background. Turn that into a point cloud.
Turn point clouds into a mesh. We turn each individual mesh
Into sequinses of tract meshes which i will explain in a moment
And the tract meshes get compressed into free viewpoint
Video or m peg file which we deliver to you.
So not every frame gets a fresh mesh. By containing consistent within
Groupings of frames we get consistent uv atlases between
Frames which allows temporal on the textures and get a much
Smaller file size which is especially useful if you are
Targeting something like the hololens.
So what is a hologram deliverable?
T file actually look like?
I'm sure you're dying to know. How do you use this in your
Application? well, the deliverable can be one
Of two packages. First, we can deliver a sequence
Of obj and png files that you are free to flip through in your
Application however you want. However, in order to leverage
Some of our special manic that we can do in our processing, we
Also compress down to m peg 4 sphoo file.
That's a pro prior tairry format but we also deliver scripts and
Plug-ins for your favorite development environment so you
Can consume these things super easily. You can see if you are targeting
Something with lower resolution that can handle maybe a lower
Bit rate on top we can set parameters at compression time.
Actually it is as we are processing these holograms, we
Can set things like texture size.
So 10k polly and 1k texture yields about 50 mega bit per
Sequence. We get that down to 24 mega bits
Pr second. That's on the lower quality end.
If you are targeting something like a samsung odyssey and you
Want a higher-quality product, we can use 80d
Pollies, 2k ses
Down to 65 mega bits per second as mp4.
The difference in quality is noticeable when you get close to
A hologram. Typically you may shrink them
Down, view them from afar. When you get up close you notice
Artifacts at the lower bit rate. For instance, here on the left,
You can see some artifacting around the player's shirt and in
The middle and lower left. Where as at artifacts are less
Pronounced at the higher bit rate.
So what platforms do we support out of the box?
Well, definitely unity. We like unity on windows,
Macios, and android. If you are using windows you can
Import using our plugin. We support ios, android and
Windows which gets you seeing kit, ar kit, ar core, whatever
You want. Now it's time to integrate into
A game engine. Integration is a snap.
Apply shade earns post process eang treat you as you woo any
Other game object. I should point out that even
Though they are human holograms, we do not support skeletons or
Other rigging used for makes. There is also limited physics
Interactivity because the plugin delivers assets, meshes and
Texture at a point in the pipeline in the gpu after which
They could be used as colliders so you could not natively use
These things as colliders. But one technique people have
Used is to preprocess the sequences into lower polly count
And objs, well, meshes, and flip through the down sampled
Objs to be used as colliders if you need them.
What does it look like if you want to use them problematically.
You use a prefab, grab the holo video object, a script we provide.
That gives you access to open a hologram and then transport
Controls like play. And that's it.
Our script will copy the hologram to native vertex
Buffers, native textures. It renders in the game like any
Other object. It has shadows and can be affect
Had by lighting. It'll update frames as necessary
While playing back. Here is what it looks like in
The unity gooing. We import the plugin.
Drag in the prefab to the scene. Position it as necessary.
Specify the path of the hologram file.
And start the game. That's it.
It is similarly simple and unreal.
I was hoping you could hear him kicking a soccer ball around.
But i assure you we also support audio.
The audio can be embedded in the m peg 4 file just like any other
Video file. What are your options when it
Comes to lighting your hologram? you have three options really.
First we can capture with con texture lighting on the stage.
For instance if you know your subject will be placed in vr
Environment where he is lit from the side, we can light him from
The side when we capture the guy.
Another alternative is to use baked in lighting from textures.
A lot of people are preprocessing texture sequences
An baking the lighting in there. You get some pretty high quality
Results. But another alternative is to do
Dynamic lighting using the vertex normals that we embed in
Our meshes at during thity rate ef processing pipeline.
That gets you some better interactivity and is certainly
Easier than preprocessing the textures or lighting the scene
As you would on the stage. Here we see our soccer player again.
This is with neutral lighting that we used when we captured
Him. Then in a different environment, including dynamic lighting from
The right side. Notice the shadows cast by his
Arms on his torso. Something that's been coming up
Reseptemberly is the issues of gaze retargeting.
A lot of our subjects are reading on the stage.
Reading off after teleprompter so their gaze is very static.
Wouldn't it be grate that if once you moved hologram niece a
Vr experience, the gaze of the presenter could follow the eyes
Of the vr viewer. Better
Interactivity from holo and viewer.
If you animate this sort of thing too far you get into the
Uncanny valley pretty quickly. But by manipulating the mesh in
The render engine we are able to use a gaze tracking pretty
Effectively. So i'm really hoping that you
Are all going to go back to your respective companies after build
And you're going to get a call one day, and your client is
Going to say, hey, i really want an asset, a video, a hologram,
Some sort of animated asset in our next game that looks like
Will smith. Now you have two options.
You can tell your animators to try to come up with an asset
That they will animate and try to get to look kind of like will
Smith or you can book some time for will on a microsoft mixed
Reality capture studio spha y capture the guy
Giving a performance that can you easily drop into your vr experience.
I think you will agree the latter sounds like a lot more
Fun. The take abies today are, i hope
You understand, that this is a rel technology.
Ready for prime time. Can you call us and book time on
The stage right now. Useable across
Ar, vr devices and the capture looks like a video production process.
They are simple and lightweight for deployment and we bring the
Value of human connection to your experience.
If you are interested in using project, please visit us at this url.
We will discuss your timeframe and definitely your budget and
Determine if it is a good fit. We hope that you will give us a kul.
We would love to hear from you. And help bring your next project
To life. Thank you.
[ Applause ] >> don't forget to evaluate this