(hip hop music)
- Okay, this might be the most impressive tech demo
I've ever seen.
It's also one of the hardest to explain on video,
but I'm gonna try.
So there's a thing called Google Project Starline.
You might have heard of it, might have not,
but Google has showed a few video examples
of this live super immersive like 3D video calling booth
where you sit down in front of it
and on the other side someone else sits down
and through the power of the internet,
it looks like an extremely realistic version of them
is actually in front of you.
It's not just a normal video call,
but it's not a hologram.
It's just, it's something different.
Either way, we got the first look at it
at Google I/O a couple of years ago.
They showed this clip in the keynote
and nobody got to record it.
Nobody even got to try it.
But now they're back with a better looking, more compact,
more simplified version.
And I got to try it and record it.
Now, just as a warning,
this first clip, it doesn't actually do it justice
because the effect does not translate very well on camera.
We'll get to that,
but I was lucky enough to get my first reaction to this,
my actual first time seeing it on camera.
All right, let's see
Project Starline. - All right,
it's gonna just start soon.
- Oh wow.
Oh! - Hey!
- Hey. - How are you?
- Great.
Okay. - Good to see you.
- [Patrick] Welcome to Starline.
- Thank you.
This is-
- Right.
- [Marques] Yeah, it's a lot more
- Yeah, you and I just saw each other right outside,
but we're in two different rooms
and we wanna make it feel like we left that space
and we came back together in a virtual space
that feels like we're completely in the same spot.
- It really looks like I can actually,
it looks like you could drop something
on this table right here.
- Yeah, yeah.
I've got a little apple over here.
So let me show you that too.
So, you know, you and I are in the same room.
Do you want me to pass this over into your space?
- This looks like
you can do it. - Here, take it.
(both laughing)
We're meant to feel like this room is just one room
instead of two separate locations.
Could be anywhere in the world.
So this is something we think is the glimpse
of what communication could look like in the next few years.
- That's incredible.
Okay. - Yeah.
- I gotta learn more about this.
Okay, so what exactly is happening here?
So I'm sitting in front of the new Starline booth
is what I call it.
Now, the first Starline they ever made was much bigger.
They had these like full room setups essentially
with a bench and computers, the display,
but also a bunch of cameras
and depth sensors placed around you.
This new one feels actually much more refined.
It looks much simpler
and it kind of feels like a reasonable product.
It's a 65 inch display on a stand
with a smaller barrier over the bottom bezel.
And then there are actually some lights on the back
that point at the wall behind the display
that serve as a key light for me, the person on the call.
And then there are no more depth sensors.
It's just a tidy array of color cameras
and microphones and speakers.
So that's at the top, the left and the right.
And they use AI to create a depth map of me
and the room I'm in.
And then the magic of this is twofold.
It's the display and the computing happening.
So the display, the part giving you
the actually impressive immersive 3D depth effect,
but then also all the computing happening to turn me,
the person on the video call
and the person on the other side,
into this realistic 3D model
just with the camera information
that you can actually look around with head tracking.
Now clearly this doesn't translate well on camera,
which is why Google has understandably been super-protective
over anyone trying it even or getting any photos
or videos of it because it just won't look right.
But I was able to convince Google to let me try something.
You see, basically once you first sit down
in the Project Starline booth,
this system has to identify where your face is
and then you can move around
and look at stuff with head and body tracking.
And of course that's not gonna work
if you just put a camera in there.
But if you do show it a face
and a camera at the same time,
like say maybe a cardboard face
where you can stick a camera lens somehow
through that cardboard face,
then it would track and you could, ah,
see what I'm talking about.
Check this out.
So what you're seeing right now is
as if you were on a video call with me
in the newest version of Google Starline.
This is the first time anyone outside of Google
has seen this sort of visualization
which is super cool.
So it looks pretty reasonable right now, right?
I can hold up an object.
You can see the colors
and the shapes and the lighting and textures on it.
And this is all information compiled
from the array of cameras around me
and the depth information comes from these regular cameras
and that's all quite cool.
But as you start moving around,
that's when it starts getting a little more interesting
'cause I can hold things out
and you start to create this parallax effect
with the head tracking where you can start
to literally look around and inspect the object
and look underneath things
as if you're literally in the room with me.
It's super convincing to the actual eye
when you see it on the screen.
Even the background behind me is not real.
It's being composited in, but I'm casting a shadow on it.
And that's being rendered in real time too.
It's a lot.
We can even flip a switch
to show a sort of a topographical map
of specifically the depth information coming
from these cameras.
So again, no depth camera's specifically being used here,
but the array of color cameras
has all this depth information being mapped
and that's what all the information
from the color cameras is being sort of projected onto.
There's a lot of processing happening here.
So it all comes together
to form this really impressive real time.
I mean I use the word immersion very lightly usually,
but I wanna stress that it's very immersive
when you actually get to use it.
So hopefully this helps you visualize.
So that is pretty cool.
But even that doesn't translate perfectly
because the feeling of depth actually comes
from the distance between your eyes.
So a camera lens is one eye,
but if you've ever tried to catch something
with one eye closed,
you know that humans perceive depth
by the difference between what you see with your left eye
and what you see with your right eye.
Your brain stitches them together
and figures out depth that way.
So even that demo doesn't quite get the full realisticness
of what I saw on camera, but it's pretty close.
The light field display in Project Starline
is literally showing a different image
to my left eye and to my right eye.
So taking advantage of that biological fact
and then letting me actually compute depth on the fly
while doing all the head tracking in real time.
It's actually kind of crazy.
Like if you've ever seen 3D glasses,
you know how when you go see a 3D movie,
like one of the eyes is blue, one of the eyes is red.
That's literally because your left eye will see
something mapped for that
and your right eye will see something mapped for that
and that's how it creates the depth effect.
But this is way better than a 3D movie.
It's so much smoother
and the head tracking and everything that goes with it
and it's, ugh, it's so good.
But then the craziest part,
the most impressive part is the computing that's happening
which is rendering the people using Starline in real time
into these 3D models.
So this is not like the metaverse thing
where the VR headset tracks face movement
and then you can have a eye contact
face-to-face conversation with someone's cartoon avatar.
But no, no, no, this is taking the actual imaging,
the lighting, the way you actually look,
what you're actually wearing
and rendering that out in real time in 3D space
and making it look like I'm talking to them
through a window.
It's kind of amazing.
The eye contact is so one-to-one real looking
and it really just comes from the fact
that I'm sitting here looking forward
and the person on the other side is also looking forward.
And so we're actually looking into each other's eyes.
So anyway, yeah, it is, the point is it's super realistic.
I promise you when I first did my demo,
the first time I ever saw it,
even knowing that it's 3D,
I still felt like I could reach out
and like high five him or like fist bump him.
He held that apple out actually.
And to my eyes it looked
like he could just drop the apple he was holding
right on the table in front of me,
which of course meant that I look stupid from this angle
'cause I'm just reaching out at nothing.
But we got a bunch of reactions
from some other studio team members
who got to join me for this demo.
I'll link a video that has some of their reactions below.
We should definitely check that out.
So it was really only the like slight glitching,
little bit of edges and fading and stuff like that
that kept it out of uncanny valley territory.
Like obviously the eye contact is one thing
and the 3D effect.
Even it had spatial audio.
So if I leaned over here,
it would sound like more audio came into one of your ears
and that was responsive.
But you know, things like in between the fingers
or like the edges of certain fabrics
or my hair especially,
some of that stuff could kind of break it a little bit
and you could tell,
but honestly I wasn't thinking about that at all.
So you might be thinking kind of the same thing I was
after I finished this demo,
which is, okay this is super cool, but what is this for?
Like what is this sort of a tech demo actually useful for?
And the answer is we don't really know yet.
So at this point in time,
Google has worked with a few companies, Salesforce, WeWork
and T-Mobile are some examples
who are literally using some of these booths for meetings,
basically, I guess.
Theoretically, that's better than a Zoom call,
although it still has its limitations.
Like the face tracking can only work
with one person at a time,
which means only one person can be in the booth at a time
to get their realistic depth effect.
So adding another person doesn't work.
I think the question will be closer to being answered
when the tech gets even better.
Like it's already gone from the size of a room
to the size of like an easel.
It could just be the size of a TV with a backpack on it.
But this will get more realistic, cheaper,
simpler and just better.
But you know, here's the thing, I actually don't know
that regular people will actually want this.
Like hear me up.
If I know anything about bleeding edge tech
which I've tried a lot of,
it's that regular people, the masses,
are not very fast to pay extra for higher fidelity,
like better quality.
So the cutoff for acceptable quality is surprisingly low
for the masses.
Like think about it, audio quality, I don't know,
streaming on Spotify over Bluetooth
seems to be good enough for most people.
AirPods are the most popular headphones in the world.
You think about cameras
and how smartphone cameras are basically good enough
for 99% of the taking pictures of your kids.
You know?
So convenience is king.
Like that's why right now FaceTime and Zoom,
it's just a square of a flat
low res video feed on a screen.
That's fine.
It's fine for most people.
When you turn it all the way up to Project Starline
which is like this incredibly realistic,
like you can see microexpressions and textures
and feel like you're in the room with the person,
that's over on one side here.
And FaceTime and Zoom are on the other side here
and this is where the masses live.
And this is like businesses, you know,
getting a booth so they don't have to fly people out
for lots of meetings overseas,
but it's still too expensive
and too difficult to get to for most people.
But that's just for now.
That's just for this version now.
I'm really looking forward to keeping an eye
on how this tech evolves from Google
and even from others who are working
on this whole 3D light field display technology stuff.
We've actually seen some interesting stuff before.
I actually tried an Asus laptop not too many weeks ago
that does the same thing.
It renders a unique image for each of your eyes
and does head tracking
for an incredibly realistic 3D effect.
But of course for that, you had to look
at a certain file that was built
in a certain model and work with certain software.
This was like rendering real people
and just having a conversation.
It's crazy.
I'm gonna thank you for Google for letting us see this
and I'm gonna hopefully be able to try future versions
when they get finished 'cause this is wild.
Let me know what you guys think.
Thanks for watching.
Catch you on the next one.
Peace.
(hip hop music)