Meta’s future plans for VR interest me, because I see them as being a sneak-peek to maybe 5-10
years into the future, and at how advanced VR devices we’ll be using might be by then.
But it isn’t just the headsets that are evolving- it’s also how we might use the technology.
And if Meta’s vision becomes a reality, it won’t just shape the future of entertainment,
but also that of work, and of human interaction itself! And Meta is one of the very few companies
capable of funding this level of research and development- so if nothing else, you get to see
the cool stuff they’re working on, and a greater appreciation for the immense challenges required,
on the road to build a perfect VR device. In the short-term, expect smaller,
incremental improvements with each generation of VR headset released-
with the next device being Project Cambria. Meta’s Product teams are always developing a
few generations of VR headset in advance. But Meta also has a Research department,
and these engineers are looking further into the future and at all the far-off problems
that they will eventually face. And it’s these projects that this video will cover.
Mark Zuckerberg believes that VR is already capable of creating the feeling of existing in a
virtual environment, but that the experience will become increasingly believable and vivid as we
approach and eventually surpass the ‘Visual Turing Test’. This is the point where VR becomes visually
indistinguishable from reality. And to achieve that, it takes more than just a higher resolution!
…although that also helps.
These prototypes I’ll be covering don’t try to cram the best of everything into
a single headset. Each device focuses on doing just one thing as well as it can.
Some of these technologies may be 5 or even 10 years away. And it would be impossible to cram
the best of all 4 into one device just now. Because the technology isn’t there.
But by building these cutting-edge prototypes, it helps give them a better
idea of what future headsets will require, and the problems they’ll face getting there.
And remember- no matter how advanced these headsets may get, the headsets Meta eventually
releases will still need to be a portable, battery powered, and affordably priced.
So obviously, Meta are paving the way for a virtual world of sorts, where you can interact
with other people from wherever you want to, taking your VR device between rooms and places
like you would currently a phone. Meta believes the magic of VR is in its ability to give a sense
of presence- and that this sensation will get better as they tackle the limitations
of the tech, and the barriers of entry. In a way, the future is already here:
more and more people are working from home. And while this brings about benefits,
it can also cut people off, socially. So you can already see how interacting with people in VR
could help reintroduce a more natural element to socialisation again. Compared to a Zoom meeting,
a VR meeting in Workrooms can be more… human. Everybody’s avatar has a place and presence in
the room, which allows you to do all sorts of things that you can in reality, like to whisper
to the person next to you, or to express yourself with your hands, or by pointing at things. A VR
space can already achieve more than a Zoom call can- and it’s only going to get better.
The first of Meta’s super-experimental headsets that I’ll be covering is the Butterscotch protype,
which prioritises a ‘high resolution’- which is an obvious benefit. But how high does a VR
headset’s display have to be, before it’s good enough? In order to achieve 20-20 clarity- which
is good enough for most people- it requires about 60 pixels per degree. For reference,
the Quest 2 manages about 20, which comparatively, can look a bit blurry and you can’t read text from
too far away. The Butterscotch prototype can go up to about 55. Which is getting very good.
At that sort of level, reading text at a distance isn’t just possible- but it’s
also quite comfortable to do so. You can also make out subtle textures on surfaces,
and the finer details on images and so on. And all this just generally gives you the level of
definition that’s required to make the world around you feel… authentic. And whereas text
on a monitor is always displayed on the same pixels, and in straight lines, when in VR,
your head is always tilting and changing its angle and position ever so slightly, and so higher
resolutions help keep text more temporally stable, no matter how shaky your FACE is.
When it comes to pixels, more is obviously better. But the technology isn’t there for
this level of clarity just yet. Due to the insane pixel definition, Butterscotch’s
field of view is severely limited, and it’s connected to, and powered by a desktop PC,
because that’s the processing power that’s required to drive such a highly detailed display.
Butterscotch can also simulate lower resolutions, allowing for Meta’s engineers to experiment and to
get a better understanding of how capable each quality setting can be. Mark is confident that
higher resolutions are the future, saying that each generation of headset tends to improve the
resolution by about 10-20%. Assuming a 15% improvement per generation over the Quest 2,
we’ll get retina-quality displays in about 7-8 generations’ time.
So how do they intend to get around the problem of rendering so many pixels?
The key thing is to optimise how things are drawn, and to squeeze the absolute most out of
the available processing power as possible. And a key part of this is to have accurate eye-tracking.
Since only the centre of your vision is in high definition, they can use eye tracking to ONLY draw
the bit that you’re looking at with high clarity, and then for the clarity to gradually fade out to
a lower resolution as it moves further away from where you’re looking at. This alone will greatly
optimise things. And by doing so, it will cut down on power consumption, which will in turn,
improve battery life and will keep the device cooler and more comfortable to wear. Win-win-win.
The HALF DOME project is all about its ‘varifocal lenses’. Because what good is a better quality
image, if your eyes can’t see it in full focus? You see, depending on whether you’re looking into
the distance, or at something close-up, your eye’s position physically changes, and the
way that VR headsets currently display things with just a single lens a set distance away,
isn’t good enough. You won’t know why it looks wrong, but it’ll look wrong. And in a way that
you’ll only understand once you try a headset with varifocals, which will suddenly feel right-
no matter how far away you might be looking. Varifocal lenses will result in less strain,
which will make prolonged VR sessions feel less tiring. Especially if you’re doing something
close-up, like reading a book that you’re holding. And the beauty is: this can all be done in the
SDK, meaning that games and the like don’t even need to know that it’s a varifocal headset,
so it should already work with everything that’s been released from, like, 2018 onwards. And it
doesn’t take users of this technology long to adapt to using it in ways that enables behaviours
that they just find comfortable using before. Half Dome was shown in several iterations
throughout the video. It started as a simple ‘moving’ lens, which could move almost 1 cm
back and forth, depending on how far away the thing was that you were trying to look at.
But over time, they’ve refined the design, gradually removing all of the moving parts
to cut down on the noise and vibration… and the latest Half Dome is all-electronic,
so there’s nothing moving about it at all- and it can refocus almost immediately. Everything with VR
is a trade-off, though, so while the electronic solution using polarised lenses to converge and
diverge the light by different amounts may SOUND ideal, the more layers you add to this,
the darker the screen gets. Which is a bad thing! They tested up to 64 levels of focus,
but settled on 32 being a sort of ‘sweet spot’, which they determined by getting people to test
the devices and to say how much they liked them. This is a technology that Mark thinks will come
out sooner rather than later, and by that, he means that you might see a headset with this
sort of technology in the next 5-6 years. Ish. And it makes me wonder if this sort of
technology could find its way into other sort of stuff, too. Like AR… or glasses.
Starburst is Meta’s HDR experiment, which aims to have super high levels of brightness and contrast,
because it turns out that the real world is much more vivid than what even the best screens today
are capable of outputting. Normal VR is about 100 nits of brightness, but Starburst can go all
the way up to 20,000. That may sound blindingly bright, but the real world can get brighter still!
…however, tests have been done with TVs and stuff to see where the sweet spot is,
and from Meta’s own testing with their headset, they seem to throw the 10,000 nit figure about
as though that’s the goal that they’re happy to aim for. So what difference does
this increased brightness make? I suppose it’s harder to describe HDR than it is to,
say… describe a higher resolution is. But more brightness gives more perceived depth to an image.
White is no longer just a whitish colour, but is instead an intense, glowing bright area-
in fact, the Starburst prototype is bright enough for you to feel your eyes physically
adapting to changes in brightness- and it’s stuff like this which goes a long way toward
aiding in the immersion of VR by imitating how our eyes adapt to real life conditions.
It isn’t just about the bright bits, though- contrast is also key, and they’ve kept the
darks looking dark by combining two different LCDs together, since using just 1 at double the
brightness would also double the brightness of the darker regions of the screen. And they sprayed the
inside of the headset black. Cos that helps too. The problem is the power that’s required to
produce this much light. It makes the device hot- meaning it needs to be bigger and bulkier
to dissipate that heat. And only about 1% of the light emitted ever reaches your eye, due to all
of the layers it has to pass through before that point! Mark says a way of improving this would,
again, be to work on the eye-tracking, and to only shine light where in the headset it is that you’re
looking, and to focus it directly through your pupil. It sounds very… invasive, but this is the
sort of thing required to increase the efficiency of a VR display, and to hit the high peaks in
brightness that are required to look authentic. So while, on the surface, it sounds like brighter
displays simply need more light, the problem is a difficult one, and Mark hints that this
level of HDR might be further off than the varifocal lenses that I spoke about earlier are.
The Holocake 2 prototype prioritises comfort. It aims to be as small, and as un-bulky,
as possible, and it achieves this via holographic pancake lenses- which are like
the next step beyond the traditional pancake designs currently seen in slim VR devices.
It isn’t just about cutting down on weight, but about moving that weight to be closer
to your head, which can also make a device feel lighter and more comfortable. And it’s
a weird experience to wear a VR device that’s THIS thin: you go to adjust its position on your head,
only to discover it doesn’t stick out much further than a pair of glasses would!
This is ideally how future VR headsets will look.
The problem is how you cram all the other technologies into something this small.
Unfortunately, these new holographic lenses are not currently as efficient
as older designs are, so they require more work before they hit the market.
A large part of the video wasn’t about a headset, but was instead about the
software that drives them. Software is easier to iterate on than hardware is,
because the development time between versions is smaller. So rather than
having to wait years to see what people think of an end product, by simulating aspects of it,
then you can cut that development time down and avoid mistakes that you’d have otherwise made.
One thing simulations can really help with is in tackling optical distortion-
that’s what this monstrosity you can see here is doing! So this simulates how displays will
look through different lenses, when viewed by different individuals. Getting the lenses for
VR devices right requires a lot of testing and tweaking for every different design of headset,
so by simulating it all like this, they can evolve the lens design a lot faster than by
having to wait for it to be physically created every time first. A lot of the distortion isn’t
terrible or immediately apparent, but it can still detract from an image’s clarity in subtle ways.
‘Canting’ is when the lenses are tilted at an angle like this to improve the field of view,
however, this introduces extra view distortion, which can be addressed with corrections,
software-side. And the distortion caused by your eyes glancing about causes real problems too,
which includes a phenomenon known as ‘pupil swim’. Again, software CAN be used to address this sort
of issue, but it requires near-perfect eye tracking to achieve it! He went on to say
that eye tracking will also help them to predict where you’ll look even before you’ve looked there,
based on initial eye movements and so on. So this helps them render the bits that are most important
with as little latency as possible. It’s all highly intricate, but I got the impression
that eye tracking is the key to progressing lots of different elements of VR devices.
To summarise: Meta deliberately over-engineered these prototypes to establish how far they have
to go to create an ideal VR device and to give their engineers a head-start in
tackling the issues on the road to realising this. As for which features we’ll see when,
they’ll be prioritised based on the new possibilities they open up,
as well as being dictated by the breakthroughs and hold-ups we face in the coming years.
Meta are already beyond the realm of using what’s readily available. So while, ideally, they’d like
to work with other companies to use pre-existing hardware wherever possible, a lot of specialist
VR tech hasn’t need to be invented until now, so Meta has to develop these technologies themselves,
which is why Mark believes Meta’s VR devices will be ahead of the rest when they’re released.
And the future isn’t just Virtual- there’s also Mixed, and Augmented Realities.
These all present their own challenges to develop, but by progressing VR, they are
also solving some of the challenges that they’ll later faces with these other technologies as well.
Will we eventually get a VR headset that combines the best of everything we’ve seen in this video?
Maybe. But maybe we won’t want that in 10 years’ time, depending on what else
becomes available or important! Mark Zuckerberg believes they’ll end up with 2 tiers of product:
a cheaper one for consumers, and a more expensive one for companies which can justify higher costs
if it can improve employee productivity even by just a few percent. But he says that these
won’t be INSANELY expensive- they’ll more likely comparable to the cost of a new PC. And obviously,
economies of scale will help drive prices down, and the benefits of one will eventually
be passed down the other product stacks. Now you know this summary, you can watch,
and more fully appreciate the full video for its deeper dive.