hair right it's known from Testim and welcome back to projections where this
week when we talking about the intersection between virtual reality
world and the real world or mixed reality one of the challenges of VR
right now is communicating the experience to you out there the viewers
you can capture what the person in the headset sees using capture tools whether
it's on desktop or mobile but the most effective way of showing people what VR
is all about aside from actually putting them in a headset is with mixed reality
recording back in the day when the first HTC vive came out there or camera rigs
are set up with ACC trackers so that you actually the camera would be tracked in
a virtual space and customs software would have to be written to then show a
mix of the virtual space and the person using the VR game in that space as well
and those tools are actually now accessible to you out there to VR users
the company called live and they have a free software tool on Steam you can try
right now and if you have like a GoPro camera and a green screen and you want
to see yourself in beat Sabre it's not that difficult to have it all set up and
to put yourself in that space that's what we wanted to try today so we
invited one of the engineers from live David over our studio set up a big green
screen backdrop a bunch of lights and put me in beat Sabre here's a little bit
of me and beat saver and also conversation with David about the
engineering what goes into that tech and what they are going to do going forward
sake listen David thank you so much for coming here and sending oh how you doing
good how are you doing really well having a lot of fun probably the most
fun I've had in beat Sabre and a long time because it's like being in one of
those commercials that is they put out because I'm in mix reality you people
seen these videos on on Facebook and and on YouTube and it's a problem in VR in
general I showing the experience absolutely yeah the whole point of what
live does yes is to help with that so King Kamea a little bit of a history of
of live and we're mixed rowdy came from back when you know people were trying to
do it with the vive headset sure so will was started a
few years ago originally our first product was just this folding sort of
green-screen cube because we have a green-screen right behind us here but it
can be difficult to set up and I screen screen like this with all the lighting
and everything so it was called the live cube it was just a bunch of lights and
stuff ended up not working out so great for us to manufacture so now we're just
focused on software I mean I you can't see it on the computer here right now
but we have our own software called the live app and the live app composites
mixed reality for you and handles calibration and all that sort of stuff
so you have sort of a holistic mixed reality creation tool I mean to try to
explain the problem or the challenge of mixed reality when I put on a headset
any of your headset most you can see what the person's seeing and that's your
fixed camera view which isn't necessarily what people out there want
to see it's it's like a it works right you can communicate that experience but
you really want the person to be in the camera frame and so you need a virtual
camera exactly um in some applications will let you put a virtual camera that
you can then put on your computer skin and record that but that doesn't show
the real you either it's not mixed so what we're filming with right now is
just a standard camera right people who have used with GoPros and pros any DSLR
SLR is exactly that humorous Oh even red cameras at very very high end you're
feeding basically that HDMI output but then how does the virtual space and the
game or whatever experience know where that real camera is relative to you in
the world yeah so the live setup starts with a calibration process and what we
do there is we sort of have a magic triangle where you take one of your
controllers first what you do is touch the camera lens so it knows where the
camera lens or the camera sensor is then you touch one corner in the top and one
in the bottom-right corner and from that it can figure out the field of view the
position in the rotation of the camera and with that information we can tell
the game put a camera a virtual camera inside the game at this exact same
position this exact same orientation and use this exact same field of view and
then once you do that you'll see that your virtual objects will line up with
the real world so I have this virtual lightsaber attached to my hand here and
then what we do from there is we have a
foreground and a background camera and what that will let us do is objects that
are behind us will be occluded you probably can't see it because of the
tracking right now and object objects in front of us will stay in front of us hmm
and that calibration that you did that we've done here that takes a place of
needing than a dedicated tracker on the camera well you still do need to
calibrate if you're using a dedicated tracker the dedicated tracker just gives
you the ability to move the camera or why are you recording right right so
when you have like those HTC vive pucks exactly those are Mountain rigged up to
a camera but that's not exactly where like the camera sensor is it doesn't
take into account focal lengths and field of view and by this triangulation
because the geometry in the game is always gonna be that that's that
triangle yes it actually does the math for all that kind of distortion yeah so
what we're doing when you're calibrating if you're not using a tracker we just
figure out where the camera is relative to your VR space so that might be your
tracking origin on a rift a vibe rift s could be you know your real world with
inside out tracking or if you do have a physical tracker attached to the camera
we just figure out the offset between the camera sensor and that tracker I
know you can tweak the you tweak V the latency and sleek the offset things to
just have everything match up perfectly because you're gonna get frames
misaligned here and there from the HDMI or from however you're capturing yeah so
all cameras have some sort of latency and usually the game is rendering faster
than the amount of time it takes for the camera to send a frame to the computer
so what we do is we want you delay the camera sorry delay the game so it aligns
perfectly with the camera feed that's just like in the frames all right sub
four frames and sub seconds so you can still play beat Sabre and play it
company has no effect on the actual gameplay
oh right right okay so you said the game is generating more views right the
virtual camera beats a Ronald box does that because they've worked with your
SDK game developers and software developers creating VR experiences want
to tap into that so that they can automatically generate those views yes
so we have our very own live SDK and what developers can do is just drop that
into their unity project or Unreal project it's super easy we're working on
a tutorial video right now and what I like to say is it takes literally two
minutes because our tutorial video is only two minutes long and once that's in
your game you can just add or you can go to the
live app here and your game will show up in the live app you can capture it and
it just sort of works magically that's really cool you mentioned occlusion as
well it's something that really blew our mind because while you can superimpose
our video image in the we see right now there's the beat saver menu
behind us when you're playing the game as we saw these video clips made by live
the the Box move behind you and actually occlude over you yes and how does that
work how does the game know where the blocks are relative to you and how does
that seamless look right now this lightsaber is cutting through me yep and
then disappearing or in front of me like it's occlusion is working right now yeah
so it's surprisingly simple what we're doing first of all is just using the
green screen or a Kinect or whatever method of background removal you're
using to segment your body from the rest of the feed that the camera sees and
from that we just have your body and we put the game right behind it but if you
just have the game behind it you'll see that objects won't go in front of your
body like they are right now yeah so then what we do is we take everything
that's between the camera and the headset and consider that the foreground
so there's a separate key the game's actually rendering two cameras once the
background and that's everything behind us mmm one's the foregrounds that's
everything between the headset and the camera and everything between the
headset and the camera will go in front of our bodies of course and then the
position of your head that's what generates that plane exactly yeah so
it's just this big clipping plane and it's relative to the camera and so it's
not rotating at all it really is just that and it can move as in and updates
in real time yep that is so cool I mean it's so much thought I mean it's a lot
of engineering work to get that to work it's it's surprisingly simple but we are
working on more advanced ways of making that work one of the things you might
notice it's very hard to actually make it happen in real world scenarios but
for example my arm maybe should be occluded here but it's not right so in
the future we'll be doing stuff like using AI K to generate that plane or if
you have a depth camera like a Kinect using the depth information to do a
better occlusion that's always going to be spot on right so is either it's
computer vision and you're using a separate get more info with a depth
camera or I cape because then you're feeding more position
full information exactly it's some knowledge about where the body my shoes
that have just this flat plane yeah but it works surprisingly well yeah it
totally works well something we were able to experiment with I know you guys
aren't making a public is also really yet to move the camera or at least
seemingly move the camera with rotation if this again blew our mind because it
gave the illusion that the camera was actually flying around although it's not
because it's not being the tracking for the camera right isn't being updated
real-time how does that work so this is something that's still kind
of experimental with us for us it's not something that we have any plans to
release yet but if there's a lot of demand for it we'll definitely think
about it more basically what we're doing with a headset like the Rif desk it's
using inside out tracking you can't use something like a vibe tracking puck to
just stick on the camera although maybe you could use some sort of external
tracking system and we have more to do with that in the future so what we're
doing here is we're just rotating the virtual world and the player just has to
sort of spin around while they're playing whatever game they're playing
yeah and it looks surprisingly realistic as if the camera is sort of doing this
big orbit around them and you can even do stuff that would be nearly impossible
to do without some crazy expensive camera track with or something like that
right and it looks fantastic as you can see in some of the footage so as long as
the the use of player can come you know can can be comfortable with that
rotating world because the actual game world is physically rotating yes and the
camera is offset from the player cameras or is in the exact same spot right oh
that is so it's it's it's really trippy to the mental gymnastics but again the
illusion is totally convincing it really is yeah you're tapping into the theme of
your compositors what makes a lot like the in in headset stuff work yeah
is there more plans because in headset right now but something I can see is I
can see basically a window with what this composite looks like right right so
while you're in the headset maybe you're live streaming or something and you
don't have another person helping you and making sure the framing is good and
all that what we do have is there's this little overlay that pops up in the
headset and right on the camera you can look at your camera and where the camera
is in real life you'll see a viewfinder as if the camera screen is facing you
except it's actually in your headset and that's what you're also using for the
calibration as well the superimpose and overlay exactly so so you don't have to
close your game at all we have an overlay and the headset that
you can use for calibration and then you just do your three taps it's just like
that and we're working on bringing more and more of our UI into VR with that's
what I was going to ask because presumably this is made for live
streamers and people want to be in VR and share that experience so you may
want other UI elements help tweak or add effects or have input from the outside
world we definitely want as much as possible controllable from it within the
headset because when you're a one-person streamer or one-person youtuber and it
does get pretty tedious going back and forth between the computer yeah live is
a it's a it's released now people can find it on Steam you have users right
download free download and people are setting up their own Queens any kind of
tips cuz like we have you know studio set up here or there actually eight
lights we have set up for even lighting you don't make that on one cue bucks
anymore but what people do to make that you know the composite look as good as
possible yes so we're working on a wide variety of ways that you can set up live
in the past it's just been using a green screen like this and this what we have
here with all this lighting and everything is definitely a very high-end
setup but we don't want that to be the only option and it's not the only option
so if you just have a small green screen and less lighting you'll still be fine
of course lighting is always great with any green screen situation but we're
working on stuff other than a green screen as well for a few months now
we've supported using a Kinect camera Trevor with the background instead of a
green screen this right here is the azure Kinect we also support the Xbox
one Kinect which is a great option because you can get them for around
forty to fifty dollars used usually and you just plug that in and it just sort
of works magically no green screen don't need any crazy lighting do you need
another camera or just uses the camera built into the Kinect so does the depth
information it kind of does all the masking in hardware and then the capture
is what you see yep that is that's really eight wow that's incredible of
what about developer relations and support I mean we saw this technology
that you guys are put out at events we've seen companies beat Sabre use it
what's legacy support like for people who've already kind of built in mixed
reality solutions using the vive tracker buck method and going forward what's the
plan yes so we definitely want as many
developers as possible using our own SDK because that gives you the best mixed
reality output using the live app but we do support older games that are just
using the steam VR plugins built-in mixed reality support
more and more developers are moving over to our SDK now we have great
relationships with all sorts of developers and more and more developers
that are coming to us asking to integrate our SDK but for older games
that don't use that we definitely support them it's just not quite as
polished and experienced yeah that's awesome thank you so much for bringing
this and setting up in our studio I think not only does it enhance
experience for the person in the headset they love you at afterward but like just
for spectators yes a little fun for some people in the room to look at the laptop
or have it than you know broadcast to a TV to see what that virtual world looks
like David's a pleasure meet you about your meaning
so that was really awesome going through that whole process and having David here
to help us with the calibration really helped me understand much better how
these systems work and they're using a bunch of really awesome tricks to
basically compensate for the fact that the virtual worlds have no idea where
the camera is in the real world and how those things are correlated it's real
cool to see that use put on using any VR headset in CPR and the live program you
can calibrate just by matching those three points and then things like
adjusting compensating for the latency differences and even kind of faking
camera moves that was super super cool and the resulting footage as you could
see just looks so compelling really fun to watch and share now they also have a
bunch of other technologies they've built into that live software if you
don't want to put yourself your real body in VR you could then actually use
their avatar system and that then creates basically an IKE a model of your
body and superimposes that over your body so you don't see you in VR you see
a virtual avatar that's pretty neat and then of course that rigging system that
I K system will then allow them in the future as David said so have even more
effective ways for occlusion so you can actually move body parts over each other
you have virtual objects you can interact with more intimately not just
swinging those beat saber blades around we also got to see a test of their
Connect system which is really right it works right out of the box with the
old-school Kinect for the Xbox one or if you want to buy the latest version for
$400 it's totally worth it because you don't need a green screen I was really
pleasantly surprised by the image quality of the Kinect output one of the
really cool things that David showed us is that with the Kinect implementation
you can actually show the beat saber blades in front of you look at camera
but the real world behind you and that was maybe my favorite use of this and in
fact if the if a developer's want to use unity and create experiences that may
not be even game really it doesn't need to be beat saber but create some type of
virtual environment cuz you can do that occlusion you can
render those views in front of the user you can basically create your own kind
of holographic interface for making some kind of short film you can do that kind
of special effect in real time with a performer in headset or not and have it
work right out of the box there's a lot of potential for this mixed reality for
filmmaking purposes and looking to the future they've already announced not
only integration with the mixed reality oculus sdk that will be coming out and
all the games I'll support that but also an implementation with iOS so you could
actually use your phone or tablet that has a our kit and use that as the camera
with the live software in all the games that will support that I think mixed
reality kind of filmmaking is really gonna be important for sharing VR in the
future and growing the VR community I want to see more developers adopt that
live SDK because I want to see myself in more experiences and more games and it's
gonna be I think another powerful tool for filmmakers and cinematographers
working in real time doing some cool effects compositing when we back in the
future with more cool technologies in the VR space to talk about thank you so
much for watching and I'll see you next time