ANJALI WHEELER: Hi, everyone.
My name is Anjali Wheeler and I'm the tech leader of 360
and 3D videos at YouTube.
So I'm going to talk about three things today.
[APPLAUSE]
Thank you.
So I'm going to talk about three things today.
I'm going to talk about to 360 video.
I'm going to talk about 3D video.
And then we mix the two together,
and we talk about 360 and 3D together.
So by a show of hands, how many of you have seen a 360 video?
How many of you have seen a spherical video?
AUDIENCE: What's the difference?
ANJALI WHEELER: They're the same thing.
[LAUGHTER]
Thanks for asking.
OK, so they're called spherical videos,
because you can imagine a viewer sitting inside
in the middle of the sphere and looking outwards,
and that worldview.
I would have liked to create one, but it's too much work.
[LAUGHTER]
So what do spherical videos looks like?
So they're essentially a sphere which
is projected onto a flat surface,
and we call this method projection.
And this method has been used for a long time by map makers,
because the Earth is a sphere, and we
like maps on a flat surface.
So this is a projection which we call equirectangular,
and as you can see, it's essentially
like a sphere, just unwrapped and flattened out.
There are several other projections
that have been used, like the world map.
That's another example of equirectangular projection.
And let's see these.
There's the quincencial on the left, and there's the Goode.
And just as an example, we want to use
things which are rectangular.
And the one on the right that is not rectangular,
it's not useful for us, because there are lots of wasted pixels
there.
So we don't use those kinds of projections.
So let's talk about how we shoot 360 video.
Today, there are lots of cameras in the market which
can shoot 360 videos.
I have one here to show.
That's a Ricoh Theta.
It has two lenses on both sides.
They both take a 180 degree view of the world.
And once it shoots, they are like all of these cameras,
have multiple lenses.
And the video streams are several,
depending on the number of lenses.
And after that, you use a stitching software
to stitch them all together into a single video frame.
So that's about capture.
Now once this video is ready, you can upload it to YouTube.
So how does YouTube know whether it's a 360
video or a standard video?
We tried several things there.
We tried auto-detection to see-- we
had a filter that will automatically
find out if a video is 360.
And the problem was, the auto-detector was very good,
but it was not perfect.
And when you go to YouTube scale,
even a 0.1% error is very big.
Imagine you uploading a 360, and it's appearing weird,
and people who are uploading standard video,
and that doesn't show up right.
So that wasn't enough for us.
Than we thought, OK, let's do a check box.
When people upload a video, they'll
tick a check box-- my video is spherical.
It shows up very well.
It doesn't work.
And we tried it before with the 3D videos.
So people uploaded 3D videos, and there was a check box,
and the problem was, a lot of people
didn't know how to use it correctly
There were people whose videos were not 3D that checked it,
and people with videos who want 3D and it didn't check it.
And everyone was frustrated, including us.
And so that didn't work well.
So we thought, OK, the best way is
to have the 360 cameras include some metadata in the video
to tell YouTube that this video is 360.
And we approached the 360 camera makers with this idea.
We created a spec for the metadata that
can be introduced into the file, and the camera makers
like this idea.
They agreed to put it in the file.
So this works.
And for a few cameras which don't have the metadata,
we have a Python script that you can run on your video.
It's not great.
I'm sorry.
But I hope that in future, with time, all cameras
are going to put the metadata in the file.
So with that, let's talk about render.
So how do we render 360 videos?
We need some sort of a 3D API, OpenGL, or DirectX
and we map the decture of the video
onto these sphere mesh, just the way it's supposed to be,
as per the projection.
And it works.
We have gravity and gyro sensors,
so when you move your device, the view pans.
Has everyone used it on a mobile device?
How many of you have used it on a mobile device?
OK, cool.
And so we call this mode the magic window mode.
And that's because this device is like a magical window
into another world, and you can move this window around,
and it moves you into the other world.
And the other is, of course, the VR mode.
So that's cool.
I sort of have some sort of a personal affinity to VR.
In the '80s, they were the first set of VR headsets
that showed up.
And they weren't really great.
They were three 20 by 200 resolution.
They had five frames per second, and they were heavy.
And nobody liked it, and so I was bummed out.
Then in 1999, "The Matrix" showed up,
and I was again revved up.
Yay, VR!
But, again, the world was not ready for VR.
So I did the neatest thing I could do.
I joined 3D FPS gaming job.
Anyways, so I'm happy that, now, we are in a much better place
far VR, and the chances of this succeeding this time
around a pretty high.
So with that, let's talk about the quality
of spherical videos.
So in this projection that we've been talking about,
there's a horizontal field of view of 360,
and a vertical of 180.
This gives us six pixels per degree.
So who knows what is the I-resolution in terms
of pixels per degree?
AUDIENCE: [INAUDIBLE]
ANJALI WHEELER: Pardon me?
AUDIENCE: [INAUDIBLE]
ANJALI WHEELER: Yeah.
So it's a 60 pixels per degree, of [INAUDIBLE] minute.
So we are at 6, we want to go to 60.
How do we go from 6 to 60?
And we are researching several things to go from 6 to 60.
So it's the eye's retina resolution.
And one of the ideas is cropping.
Let's crop the top and the bottom,
because people don't look up when
they are watching the 360 videos,
or they don't spend time looking at the floor.
They look forward.
So how about we crop them out, and then we
get close to ten pixels per degree.
The other ideas are tiling.
Let's have several streams, and when people turn their heads,
you switch to a new stream.
And dialing gives us another advantage in horizontal.
We get almost 10.6.
We you combine them together.
We get 10 on both sides, approximately.
And 4K [INAUDIBLE] are just around the corner.
They're right there.
And so that takes us to 20.
So that's good.
We are close.
So that's one of the things we're trying
to do-- improve the quality.
And another thing that is tough with the spherical videos
is just the sheer amount of data.
So if you look at this image, the area under the red
is the area captured by a standard video.
And we have like four times more things there.
And add depth and 3D to it, it gets even worse.
So that's another area we'd like to improve upon.
So that's about 360, and I'm going to talk about 3D
after this.
So are there any questions so far?
Yes, please.
AUDIENCE: [INAUDIBLE]
ANJALI WHEELER: Yeah.
But those features, we are doing on a, you could say,
platform-by-platform basis.
So on the web, we support zooming in and out.
And on other clients, we do wish to.
And I cannot say when we'd do it.
So let's talk about 3D.
It's sort of an old story, but I feel
like I should talk about it before we talk about 3D
and spherical together.
And so how do humans perceive 3D?
This is a perfectly 2D image.
And I'm pretty sure everyone of you
here can figure out what's far away and what's close by.
So human can completely make sense out of a completely 2D.
You could see a picture and know what's far, what's near,
but we know that this is fake.
This is not real depth.
And so we perceive 3D by something by disparity.
We have two eyes at two different locations that see
two different use the world.
And we compute the difference, and we figure out what's far
and what's near.
So that's what we try to replicate
when we do 3D in computing.
And
one example is, there are 3D cameras, which
just have two lenses set exactly the same distance as human eye.
And they capture two views, and we can just
put them side by side.
We can put them top and bottom.
And people have tried every possible way
to do it-- interlaced and temporal alternate frames-- one
for the left eye, one for the right eye.
And all of them produce, more or less,
similar results where there's a viewing instrument that people
wear to filter the left and the right eye correctly.
So we can always watch a 2D view of 3D
through one of the eyes' view and just watch it in 2D.
This is the anaglyph, or these red and blue filters.
And we could polarized, which most movie theaters do that,
and a lot of TVs have organized glasses support too.
One downside with polarization is
that it cuts the brightness in half,
so the TV or the movie theater has to be twice as bright.
Then we have VR.
Cool.
So that's about 3D.
So before I talk about spherical, any questions on 3D?
No.
Cool.
So now we're going to mix the two together.
We talked about top-bottom.
We talked about equirectangular.
And we mix the two together, and we
get equirectangular top-bottom.
And that's the most popular way today
to represent spherical videos.
How do we capture them?
JUMP.
There's a talk right after me.
You'll learn about JUMP there.
So let's talk about playback.
Playback is, again, two of the same idea.
There's a magic window mode, and there's a cardboard view.
And that's how people will be able to enjoy spherical 3D.
We don't support it yet, but we do plan to soon.
And so that's rendering part.
But when we introduce 3D, there are
a lot of things that are needed to make
the users comfortable-- like, some people
get motion sick when they watch VR,
and some people get headaches.
And so there are some things to keep
in mind, for example, the IPD, or the interpupillary distance.
So humans come in all shapes and sizes, and so people whose eyes
are closer together, they are not very comfortable
when they see VR, which is made from the point of view
of somebody whose eyes are farther apart.
So it's, essentially, the head size.
And so we need to make sure that the IPD is
right for the person, and maybe provide a way
to calibrate and adjust for it.
They are field of view issues.
So there's been research done on it,
and it was found that people are more comfortable when
the field of view is hire.
People also get motion sick more when the field of view
is higher.
So there's no good, great path there.
It's a balance act.
And the third thing is latency-- like when
people move their head, how long does it
take for the image to move?
And that's, again, something that
needs to be as low as possible.
So there's the tech aspects of 360, 3D, and the YouTube part.
Thank you.
[APPLAUSE]
[MUSIC PLAYING]