all right hello world this is cs50 on
twitch my name is Colton Ogden and today
I'm joined by cs50 s Connor Doyle Connor
thank you for coming under this tree
thank you what are we what are we
talking about today
yeah so welcome everyone excited to be
here on my first time on the stream and
today we're going to be looking at
augmented reality development
specifically in spark AR which is a tool
released by Facebook to develop
augmented reality applications on
Android and iOS devices um on messenger
Instagram so yes I'm pretty excited yog
materiality is definitely trending you
and I were just talking about this week
before the stream it's been around for a
few years but I think you know given
hardware limitations probably wasn't you
know in its initial phase probably
wasn't as popular as it's surely going
to be soon tools like spark AR which
you're going to talk about today are you
know sort of what we need to be able to
progress in that sense yeah I think
right now or even augmented reality it's
like a really never less time this mixed
reality there's virtual reality there's
augmented reality there's X are all
these different kind of like vocabulary
for describing this kind of movement
into the virtual realm where we're going
to experiencing sort of the physical
world in this kind of digital way and
that kind of hybrid is in between the
digital and the sort of physical is kind
of what AR is explore anything and
snapchat obviously have been a big
proponent they've released the similar
software the spark IR and a lot of
different unity lot of has loads of AR
applications not just for a mobile
device but also like hololens magically
different things like that so there's
definitely a lot of excitement about a
alright now in its application or mr and
its application in the world but i think
you're right i think like simple tools
like spark ir which will allow you to
simply simple in the term that is really
easy to get into
they're quite sophisticated but simple
to solve get grips with allow you to
sort of explore the kind of storytelling
and sort of possibilities of this medium
for future interaction because I think
that's what it's gonna be I think right
now like everyone's looking at these
devices going oh it's really hard I was
really difficult but
we've got a point in vision away in
which it's going to become very seamless
with my interaction with the world and
Mike one way just begin to visualize how
we're gonna interact like in UI design I
know people use spoke out for like UI
different things like that like one way
to start to explore or play with those
possibilities is easy that's all like
this so right
yeah we're only as powerful as the tools
that we have yeah and you've sort of
been at the forefront of you know ever
since the VR phase sort is good I think
it was 2015 cry yeah you've sort of been
kind of passionate about this sort of
new progression that we're taking in the
world of tech yeah and this is sort of
how you came on to cs50 would you mind
talking maybe a little bit about how you
got introduced to C's 50 yeah your
history with us so far sure yeah so so
back when I was a freshman I'm a college
student at Harvard College on the
studying means literature my senior and
back when we was a freshman I was really
interested in like storytelling
specifically looking at kind of like
theater and how theater could scale to
different places around the world or how
we could give a sense of like theatre in
different environments and theater like
not in it not in an actual traditional
setting so that led me into virtual
reality actually I went along to event
in Boston on that kaleidoscope held with
Rene and it's kind of like a film
festival but for virtual reality
experiences and I tried on a headset was
immediately like well blown away and was
like this is definitely going to be the
future of like how we experience the
world how we relate to it there's
something there's a seed in it that's
kind of special so that began my journey
into VR and then I I was looking at sort
of coding things on campus and I came
across cs50 and I reached out to David
and we began talking about sort of some
of the video aspects of what's been
happening at cs50 obviously incredible
video production team and 50 production
and everyone but I was like you
what about via what about virtual
reality in education what about virtual
reality in cs50 and that's when the
conversation started in
a lot of really cool projects and
developments we did cs50 around the
world which was we sent sort of via our
cameras movie our cameras to different
people around the world who takes it if
the online we got to see their world in
environment and with films sort of
twenty two lectures in virtual reality
as well that when alongside the actual
traditional capture so that people could
put on a headset and be at home and it's
see David and see lectures so and that's
really where my sort of journey with c50
started and it's been developing
evidence there's a very normal project
and I think we may have been the first
University to film there like we love
yeah yeah pretty amazing pretty amazing
accomplishment but yeah I totally agree
I think that VR and AR have a lot of
potential and especially once they
proliferate it I think because right now
it's not common I think that you see
people with it with the actual hardware
yeah but you know more much like
computers themselves back in the 80's
90's as soon as we actually proliferate
this hardware we make this a
commercialized thing yeah to the extent
that computers are smartphones yeah I
think you're right I think we have a lot
of crazy awesome stuff sort of the near
future and this today we're talking
about spike there's kind of a step in
that direction I'm gonna actually switch
over to your screen here if show all set
here and also why don't we plug up a
question we had a hole so sinister rules
a great accent shout us to Connor for
being our first UK representative on
string ah but Andre asked XR what does
that stand for yes so XO that's a great
question X is the coin of so because
we've got augmented reality virtual
reality mixed reality sort of XR is kind
of sort of this the space between my
okay we don't really know what the
reality is yet it's like all the it's
like all of them combined it's sort of
like all these kind of X nebulous stuff
reality it's like an umbrella term so
umbrella return correct um peaky
blinders is also quite funny that's a
that's a great reference I have three
months ago
good accent pick up there yes so but no
to what you were saying I think that in
terms of Forex are at least because of
the definitional difficulties with this
technology I mean like it's moving so
fast and different hardware providers
are trying to scramble to try and create
these kind of lightweight glasses which
enables sort of this kind of mixed
reality experience always it all mented
reality or is it holograms or is it
projections this is really kind of
difficult to kind of disown actually
what this stuff is because in the
popular language you know people might
say a hologram is I think is actually
projected into space this material is in
space and that's like what some
definitions prescribe it is that and
then other people other you know
hardware manufacturers will say that
actually if it's projecting into the
retina then maybe it's just that is a
hologram as well if it appears to be in
space and it's a hologram so to get into
difficulties different interpretations a
lot of different versions of the stuff a
lot of different deployments of this
idea of accentuating the real world with
things that are not physically there
correct so I have a visual sort of
representation sure oh not just visual I
mean augmented reality can can be
basically anything that's adding to the
reality so it can be sort of haptic it
can be auditory it can be sort of taste
smell all that sort of stuff and there's
a really cool developments actually
happening in like unique sense that
being you know generated you know in a
are and like weird stuff like that
that's funny I remember back in like the
90s I think there was a like
smellivision yeah eighties well though
this was a thing that people tried out
obviously the technology back there was
a lot worse probably than it is now so
I'm curious I would be very curious
because I think that would really take
people in different great I specially
with games yeah I imagine going into
like the first thing that popped into my
mind I've been playing wriggle - like
there's a sewer level I was going into a
sewer level become a grave absolutely
disgusting I think about Charlie in a
chocolate factory where you can like
really might take the chocolate bar out
the screen actually smell it yeah yeah
and hey I don't know how feasible I mean
I'm all factory and sort of tastes are
related so I'm curious like how that
would evolve into like taste feedback
yeah if there's even a possibility
yeah and looked isis-linked the smell
quite a lot as well
there's kind of these interesting
questions and these are all going to be
potential realms I mean it kind of seems
far off right now like oh well I'm gonna
be tight
would it be like smelling stuff in gay
people who are exploring it and you know
maybe it's kind of conical men yeah
Andre is interesting question the actual
reality has problems with causing motion
sickness how does that work with AR
that's a great question so one of the
reasons virtuality causes motion
sickness or has been like said to courts
motion sickness was that because
basically wearing a VR headset it
occludes
it stops you from seeing the world at
all so that if your body's moving and
you're not moving in AR I've been in
virtual reality sorry if you're it's
just so if you like walking around but
the headsets not tracking properly then
you have this weird out-of-body
experience where like the track kind of
throws you and you kind of feel nauseous
whereas a are because AR is projecting
onto the world so it's taking the sort
of given sort of visual plane as a sort
of base level and then just adding
certain things on top of it
then you're not going to get the same
kind of total out-of-body experience
because you always have a reference
point in the world so even if the track
doesn't track properly it's just gonna
be like jumping around they're not gonna
be like actually throwing you're kind of
whole body around in the same way VR
those so that's one way in which motion
sickness is not as much of a problem in
augmented reality we don't coming over
here just yeah little betting you're
getting cut off there apologize for that
all right awesome so AR will make
millionaires so on that note so spark AR
so this is what you have loaded up where
can folks if folks want to get spark
after salt is it free yeah and where can
people download it if it is yeah so
Facebook release is completely for free
and you can download it right now and I
can get up get it up
it's ba K or com just download I think
it's back to 250 megabytes or something
and you sign in with Facebook and
essentially allows you to deploy these
whatever you develop in the development
environment directly to Facebook and you
can share with your friends on Facebook
and there's a closed Instagram beta as
well and now as well that you can apply
for so you can also release filters
directly to Instagram
and one of the coolest things I think
about this and one of the reasons why I
really enjoy using this tool and
advising others on using this tool is
that the seamless quality of it for user
experience so say you built this
incredible augmented without an
application in in unity or unreal and
you're like hey I really want to show
you this app we want to show you this
like experience
download this app and like that's been
one of the biggest problems like the
friction barrier they're like having
people having to like go on download the
app just experience look like a simple
filter or something like they're not
gonna do it especially works well for
games I'm guessing but not or something
that someone's gonna spend ten seconds
exactly
they'll delete the app instantly so why
Spock I always interesting because it's
literally just a link to if you have
messenger installed which so quite a few
people do Instagram then it just loads
the camera in that in that service and
just allows you just use the Spock I are
sort of creations directly in the device
with that app already loaded so that's
one of the best things about anything is
there is there the ability to make
standalone apps with Sparky our studio
as well as are intended to strictly for
attendee strictly for Facebook picture
yeah you deployed directly to Instagram
yeah that makes sense I mean a plenty of
people use it so it doesn't really
matter too much
yeah that's cool all right well I'm
excited to learn a little bit about it
cuz I don't know almost anything about
this right okay so should we dive in
yeah they've been doing it and if folks
have any questions while we're doing
this definitely post with a chat I'll
take a look yeah smelling games is a
terrible idea I bet it'll be killed by
pull my finger jokes in fortnight from
pull my finger jokers and Fort mrs.
Andre maybe I would like that I say be
kinda grim yeah so let's yeah let's slow
do something that's fighters put me up
so I'm just gonna quickly explain like
what we're looking up here before before
I stopped building anything intake
talking about the features so this video
always makes me laugh so this is a stock
video it's a stock video and you'll see
why the heads tilting and stuff in a
minute but so what we're going to be
focusing on is there's spark eight are
basically allows you to use both the
front and rear facing
I'm in on a phone and basically how it
does that is we use as a messenger app
but for the front-facing camera are you
the one that's facing a face like the
one that's being shown in this sort of
portal view window right now is there's
different things that it it sort of has
built in it has like a face tracker to
like perfectly sort of track the face in
3d space it has a hand tracker and then
for the reverse camera so for the rear
facing camera that faces off into the
world if you can it sort of take a
picture of something it has a plane
tracker so a plane is basically any kind
of surface and what why that's important
is because if you were to place an
object into the world or at least and
make it appear to be in the world then
there needs to be a sort of stable
tracking surface that the phone can sort
of know the position of in order to
accurately place that digital asset into
the physical environment so that's what
rear facing camera does front facing
camera is what we're going to be looking
at primarily today and specifically
we're going to be looking at some of the
face tracking and face mesh tools that
are built directly into Spokeo
cool cool somebody ashley i sent a
pretty much any question kind of
suspected the Brit accent thanks to the
series / movies but I can't distinguish
between Scottish Brit accent you have
any tips for distinguishing between a
Scottish no British accent can you do it
can you do a Scottish accent Scottish
accent what I'm sort of I wonder if it's
a stereotype maybe the UK people all can
do each other's accents yeah we call it
we can then the Scottish accent is like
it's definitely distinctive in fact it's
kind of distinctive we have loads of
different dialects in England that very
own region even like Birmingham versus
like Newcastle which is best-laid you
know that's a major one knows Luke was
looking at Newcastle yeah yeah so
there's loads of different accent I
don't know maybe watch Braveheart
that's true that's true
okay so great so wait so we have
different couple of panels here let me
just talk you through so we have the
inspector panel on the right hand side
which has any item that you've cooked
you click on it so it brings up the sort
of features that you can change or
different parameters on the right hand
side I should do I should probably get
rid of our chair get rid of chat it's
like a little bit a little bit hard to
see yeah I'll just get rid of it all
together there we go they go like that
for a sec yeah so on the right hand side
here you have this kind of inspector
which has all the sort of values that
you can change depending on what you're
clicking on in your scene over here in
the scene panel you have the device then
you have the camera of device and here
you've got the front and back camera but
you can select what currently selecting
on the front camera and we must have the
microphone as well I think you do like
sonic can like you can do things based
on audio yeah as well yeah there you can
play sounds but you can also distort
sounds coming in as well see my legs
right yo and make something happen
yeah that's pretty cool yeah that's
feasible um so in the camera I said so
if we look in this scene right here this
is kind of a 3d sort of like unity or
months like it's almost like it's an
that looks a lot like unity be is very
much like unity in terms of this and
there's loads of different views as well
you see you've got like that was
perspective but you've got orthographic
you've got bird's eye that's right
sort of front these are all different
views of the 3d scene that's cool yeah
yeah very much like a blender or unity
interface correct yeah we're company
from the front say give me a second
how did we just okay once they get here
but no so we can see it so here we've
got the face and this is just looking
directly down the camera so the first
thing we do is going to show you the
face tracker unison so if we insert here
for click this plus button up here is
that is that cutoff in this oh no this
just happens to be windows so if we
insert a face tracker it looks like
nothing happened because we don't have a
mesh to render that face track of them
right now the face tracker is the sort
of thing that
is tracking the face in the scene but in
order to represent that track you might
want to insert sort of face mesh so by
hitting face mesh well now we can see
how accurately it's tracking the face in
the scene and some people are asking
yeah why is the woman in the shop making
all those facial expressions
yeah but presumably to test I'm assuming
to test different things right yes so
the reason why that there's kind of this
weird looping video looking left and
right smiling is because you got if we
now look at that mesh we can see how
well the face measure is actually
performing on and tracking on the face
you can just especially see it when she
opens her mouth when she does like the
shocked look yeah and also if we look up
here on the inspector panel we have
different things it's for this mesh so
we can actually disable the eyes oh
information we can also disable the
mouth as well so that's kind of weird
why might someone want to do something
like that so say if we so say for
example like we want to create a mask on
the face which I'm going to show you in
a minute by applying some sort of
material to this face mesh if we want me
to the ice to be visible in the material
or if we wanted the mouth to be visible
invisible this is just a very easy way
to do got it makes sense yeah so this is
pretty simple this is just a face
tracker but it's quite a sophisticated
it's built straight in and another thing
I want to point off right at the bat is
that this person is doing all these like
weird funny faces but if I go to video
up here I can change the person doing
the video it's different a couple ones
at Facebook if so will give you here or
we can actually hit the FaceTime HD
camera and pointed at me no surprise
with a green screen behind us take it
out later I am you can see tracks pretty
tracks pretty well and it's all just
happening in real time this is all real
on pre-recorded video this is you
actively right now making yes to me my
conceived Isis yeah Andre is asking
won't be the primary device for using an
AR app oculus or a mobile phone tablet
yes so again just to reiterate that
spark AR is
spark AR is built for Facebook
applications so it's built by Facebook
it's meant to be deployed on Facebook
services so right now it's deployed on
messenger and Instagram in closed beta
so that's what this kind of application
is built for so particularly mobile but
it can be on iOS and Android
sure yeah works really well liberal
thing it works great it doesn't a
collector really well um if we go into
assets so now we've got a face mission
we've got if I put my hand you can see
the kind of distorts a bit kind of needs
to detect the face we've got this face
mesh going on if I want to apply a
material to the mesh instead of this
kind of canvas if I learn assets and I
hit create new material and then I on
this under the face mesh there's a
material box here if I click add and I
hit my default material now I've got
material on my face so I've got this
kind of very simple very plain white
sort of default material and I can
disable my eyes which is kind of really
weird as I was saying right yeah so this
would be a I guess if you had a texture
a model that you didn't want to see your
eyes through yeah it would make
completely disable my mouth which I just
did as well well we can just set the
whole thing we get the real deal so I'm
gonna keep the always for now in the
mouth and then there's loads of
different why this is so cool is that
you have this kind of assets and the
device tracker is that under the
material shader under the material sorry
there were those different like things
you can apply to the material is a
shading options so under diffuse you can
apply sort of different color different
colors so that to the mask pretty simply
by just like positioning it in different
ways it's pretty simple
um you can then almost really get off
that so to go back to white the other
thing you'll pick the robot game is I'm
assuming they're talking about mr. robot
oh oh maybe in my robot there's a
there's a show called mr. robot oh no oh
yeah I right I know about that yeah
I think the party I'm talking about yeah
there's really cool stuff like specular
which can introduce the kind of
shininess to the mask in different ways
and then if I were to like this is why
it's so good you ended just playing with
this other stuff to take down the
opacity a bit you start getting these
kind of really conic interesting weird
effects on the mask right now there's
like a hot I immediately what like want
to solve this right now there's like a
hard line on this mask at the top all
right yeah and so I wanted to get rid of
that hard line I'm gonna make it look
like it's more of your actual face yeah
it blends it a bit more there's
different couple ways you could do that
but one way actually it would be to
enable this thing called alpha down here
and under the under the texture part of
the Alpha spark I'll give you face
reference assets up here in a folder I'm
just learning the mask like actually
pretty well lip-sync to what you're
saying yeah no at all it's real time if
I go into texture and if I choose now
under the reference face assets which I
just pointed out it's under the Help
section of sparkle and then if we go
into face mesh mask and if I open that
you can see now this is one blend right
top of the mask the color is a little
bit so right here you would maybe want
to change the color of a two to match
your skin tone I'm guessing yeah always
gonna actually show something even more
interesting in a minute thinker can
apply this is just again this is just a
default material this also you can apply
image textures here you can apply
external textures which are kind of
interesting so you can pull textures
from a URL that's cool that's hosted
online over there changing or you can
also host videos as external textures
and you could actually video put a mask
video directly onto this mask
I'm sure Manali was asking what webcam
are using this is just a front facing
webcam on my Mac you know just a Maximus
macbook webcam webcam and if he's asking
can we have a different voice I'm
guessing in the software but that would
be kind of hard to I think show on
camera yeah I don't even think Chinese
on camera but yeah you can I think you
can definitely distort the voice in in
that
I think they're always of digne for a
member kite yeah we don't have an audio
setup for his computer currently at the
moment but great so we've got this weird
mass going on let's kind of let's get a
bit more into the features of this tool
set now and think about you mentioned
like not having my face let's kind of
explore deeper into like potentially
getting my face onto my face in this
kind of a weird way all right
sorry boy so I get rid of this material
for a second
and we're back to our favorite sound
leakage on this track match exactly
under face tracker they actually have
this really cool tool called texture
extraction and if you hit text
extraction it creates a material down
here called phase tracker the number of
the face tracker which is from the
currently face tracker zero face jack is
zero texture it's a boy go and create a
new material now and under the material
under texture I apply the face trap is
irritating or just made and then if I go
to this is just for the material now and
we'll call this material face I'll call
it my face if I go into face tracker
interface mesh which is the sort of
canvas thinks it's highlighted in the
the window now and under material if I
add my face bang here is my face is if
texture being generated of your face
that's getting applied to to my face
I've mapped to the mesh and you can kind
of see this a bit better if I would like
to insert maybe like say a rectangle or
something real quick and if I were to
like say if I were to bring if I go into
the face tracker and I bring the face
mesh maybe off my face slightly look
forward
oh right yeah okay um I don't even need
a rectangle can you see that you can
kind of go where like there it is you
can kind of see how that's like how
that's kind of crazy
so it's following your face motion and
it's applying this transformation to
three object that now has it go to see
what it now has texture
of your face onto it yeah and like right
now it's a bit dark so if I were to
under the Handy lighting here forward to
like apply another ambient light into
the scene you can see it gets a bit
lighter now it's kind of a bit more
matched and then maybe I want to like
maybe I want to bring down
maybe the my face material is a bit too
visible right now maybe want to change
the opacity of the material of it oh
into Christmas cool like power dog thing
so you can see already like just by
playing with this is some really
interesting things you can do we bring
me a pasty back up here like we go into
the face track or going to the face mesh
we can see we've got noise and mouth but
now if I enable the eyes you can see
here my eyes back on the content texture
and here's my pan there's my mouth gone
and there's my eyes gone so you put that
back onto your face now I noticed that
there was like a sort of a difference in
skin tone between the texture and we
assume here that might be a lighting
function that is a lion function yeah
and you can if I want to drink screws up
it isn't it doesn't recognize it that's
funny
yeah but already you can do pretty cool
stuff and Mike what a simple tool a
simple thing that I really like starting
to do from this was like okay so if you
just pause the tracking for a minute
we've got one Facemash wait but there's
no reason that only needs to be one what
if we were to duplicate the face tracker
and the face mesh and like maybe offset
that in a different in a different plane
so if we white no actually this is kind
of annoying is there like a like a right
click to rotate the scene or something
yeah this should be I don't know why
that is in there we go nice alright if I
go back to perspective here now okay so
if I if I click on the face mesh you can
see that one is here and the other one
is for next to it before I move this
back now if we hit play
so thanks to our Kedah ratio for the
fall appreciate a great now you can see
we booked two to two your face is doing
it nice this Pat one doesn't have in my
face on it but I can pretty easily under
face so just you just duplicated the
mesh that was already there I just moved
it further why just into the mesh and
now I've got to like minimis the
operating as you can see relative to
position to my face
so you can have like a scene with like
five different people booked with the
same face all of them that's one
potential thing you could do it
correctly um this is kind of looking
pretty crazy there right now have we got
alpha on this so maybe if I just put if
I go back to face mesh mask and I've got
this kind of interesting
alpha blur remember the thing we did
earlier on the blur oh right yeah yeah
to create that he's only more seemly
before you see this is hard line right
and uh if I redo that
now we're blending again so it's kind of
really fun and like um this is kind of
like kind of a quiet advanced I say
feature of it but to track faces is
something that I really like because I
think that how it's put is consulting
one thing you think it would do what you
think it's doing so it's kind of really
the hypnotic thing to do this just like
way to say one thing you think it's
doing is you think it's just pulling
that the camera data and applying it
directly onto the face and I tried doing
that because you can also pull but the
actual camera feed and put it into
objects in the scene and it's not doing
that because if you think about it if it
takes a flap if you take a flap sort of
thing of a face and put it onto a mask
of a face it's gonna distort and not be
mapped correctly to the face it's
actually working out the geometry it's
using the 3d days it's kind of reverse
engineer you have an image of what the
face book yeah well it's unreal and I
can show you so under reference face
assets here and under textures you can
see that like this is weird but you can
see like this is what a texture might be
that you would apply to the face
facebait this is what it essentially has
to generate and then wrap around this is
what is generating and what it's
wrapping back on to your face from a
standard feed which is kind of cool
yeah so here's a certainly more muscular
ruse and muscular sort of faint
masculine it says here face and then if
we go into this you can see the solji oh
of the face mesh and then that was the
face mesh mask which creates a kind of
yeah exactly it's like a UV unwrap yeah
and then here's the the tracking points
on it so if I wanted to create like a
really cool like face paint or like
there's lots of different people of the
nice effects whether they've got like
face paints or things of like anime
around the face and stuff they'll use
kind of something like this as a
reference set and I'll bring it into
Photoshop and now sort of distort it all
or you know graphic image manipulator
program or whatever and they'll distort
it in whatever way they want and then
when it wraps onto the face it will wrap
correctly onto the face I'm doing is
asking at the top left is of the light
source for the shading back in the in
the studio this yeah this is the
directional light so so when you move
that around you'll see changes in what
the other models ourselves let's light
it let's just get rid of let's just
actually just go back to to base on this
and we'll just to reapply a face tracker
and a face measure up quick because I
also want to see it with the blending -
let's see how convincing we can make it
look so hot on so if I just bring this
back on I think I need to reset it to
zero
okay so this is on my face what we say
um well a couple things that I would
like to see first off use the blend
thing that you talked about now while
it's on your feet like first of all get
the image back on - it is in my face
back on yeah yeah okay
so that that would be going to my
material and my face right and then I
don't think that's is it hmm it's
tracking correctly here it's kind of
interesting one second I was gonna say
if there was actually the mesh that no
that's why you do because if you hit
material my face and they never go to my
face and I hear face tracker zero
texture kind of interesting one moment I
think it's cuz I deleted too many of
these I kind of just run the deleted
stuff so I think if I were you I
recreate it so just to creeping back go
back over this and speed face tracker
face mesh going to face tracker texture
extract that face tracker texture here
create a new material call it whatever
you want we'll call it like my face or
something no going to the my face under
texture clicks face tracker zero texture
down under face mesh material we're back
take a while here we go so now we have
you know what we were at before and now
what if we applied the blend operation
to that - what the Alpha thing yeah yeah
I know we did yeah we did do that
earlier but we'll do it again alpha
under choose file we go to face mesh
mask and there that's that that's pretty
of nice yeah see there little so you
look more convincing actually matches
the skin tone pretty well yeah it's a
bit dark and one of the things you can
do this isn't the only way you can do
this but you can start if you've got
your own face um and if I take that Muir
pass there we go that's pretty good yeah
as the you considers a bit of failing it
but you can start doing cool stuff like
there were other ways to do this but you
can start when it fired specular now and
I really if I you can see that it's not
applying to a shake is applying to the
actual skin tone really to a festival or
something there we go is there any way
it moist people love this kind of look
you mind if I test it on mine as well
yeah go ahead
let's see what you look like this it
will work yeah well oh wow look at that
look I used to put on some makeup or
something that's funny yeah it's pretty
crazy how that we're excited and then it
screws it up without oh wait I know
oh is maybe I got used to yours or
something and then obviously you can
what what's cool is that you can start
applying like colors and stuff like
because it's got the face now and it's
got the GL of it if you change this look
so ridiculous
also David popped in the chat saying
these are you gonna see Connor Connor
Connor and Colton the reference to your
three feels like a decline iron core so
this is kind of weird but you kind of
get the gist you stop playing this tool
and you stop going there's so many
options here to to do things um this is
tracking on the face it doesn't even
look like it's tracking on my face but
you can see here in this panel that it's
doing a pretty pretty good job of doing
it another thing is that white under
face Chapel right now but we have a face
mesh but that is not the only object
that can be sort of truck a trap to a
face if I wanted to sort of like and for
important asset right now like there's a
project I did I really like the design
of Virgil hablo actually and as a sort
of funny thing I thought it would be
great if I sort of took the famous
speech marks of Virgil hablo and tracked
that to my face so what I'd created was
um in Photoshop was um a 3d
I extruded some text and corrected a 3d
object you can also download through the
object online um buzzer this quotation
mark these are just quotation marks
right with some space in the middle and
then in spock ir this was kind of the
first thing i did actually was to test
it was i just drop these in and if i get
rid of that oh yeah if i get rid of the
face mesh now and just have these you
can kind of see that these are kind of
traps to my head okay in this kind of
funny sort of way
see so if you start thinking through the
possibilities of what this means is like
you can you can recreate like artistic
you can recreate any sort of artistic
sort of portrayal or anything using the
face not you don't have to like just use
the face as a face you can use the face
as a sort of reference point for other
objects in the scene yeah you just at
this point you're just taking three
objects and manipulating their position
orientation yeah and you manipulating
the position orientation relative to the
space there's another feature that i
wanna get into before we go deep into
the face track although we've done quite
long the face tracking now is background
segmentation so another big another big
and i'll take another big feature or
actually another big feature of spot
they are one of the and we haven't even
scratched a surface really and this is
meant to be a sort of introduction spot
or not by any means an in-depth
exploration and there are tutorials and
resources online of really really good
so i definitely check them out if you're
interested in any aspects of this at
spoke a calm but here right so say we
want to do a background segmentation
what does that mean so right now we're
actually being green screened as we saw
earlier Burroughs surprise were being
keyed on a green screen and that means
that the camera is looking for green
pixels and then it's taking those green
pixels out of the image and replacing
them with whatever background they want
in this case my laptop screen well you
cannot do that without the green screen
anywhere in the world using like
using your front facing camera and spa
Caleb this isn't all the tools do this
and I'm going to show you know how we
can do this on spa chaos specifically so
in spock they are under camera which is
sort the top left the camera lee it's
sort of built into the seed we have
texture that we have segmentation here
now if we hit plus then we hit person on
segmentation we craved down here on the
textures we've created a segmentation
mask texture so if i go into the camera
and i insert like a background so like a
rectangle say and then i make that the
size of the screen by hitting fill
pairing we now have this sort of canvas
as the background but the problem is
what how does it know what to cut around
the cuz right now is just overlaid
directly in front of the camera we want
to create we want to kind of cook around
the person and what might do that is
using this segmentation mask a texture
that the camera generates automatically
for you ever see and the way to do that
is to under the rectangle we need to add
a material so we're going to create a
material and we're going to call it
segmentation segmentation background
under segmentation background if we then
and we're going to go alpha click alpha
and we go
segmentation mask texture and then we
hit invert let me go back to that was
just sitting on the material itself and
then we need to apply the material to
the rectangle we hit rectangle and we
select the material and we see
segmentation background here you can now
see not too clearly yet but if i apply a
color you'll be able to see it that the
background can be anything we want now
and it's being segmented that's awesome
we have some sort of algorithm I'm
guessing that like can detect or which
you know what part is the person front
but parts the background yes
it doesn't it does a pretty good job
like if you see that that's key yeah
it's not bad
no no pretty well and you know if we
changed this just to prove the point
here like this isn't the color of the
wall this is like the color that we
selected on the color wheel how old does
it deal with like background objects
like if I had if I just pretty I mean it
does pretty well I stood behind you when
it's still you stand behind me yeah oh
so it actually okay so it takes people
it takes people yeah I disappeared you
dissipate that for a sec so yeah I'm
like as you can see it's calling if you
look close its frame my hairs hairs are
always difficult to Kay right yeah
because colors gonna seeps through them
a little bit yeah but if we go into
segmentation if you're going to the
segmentation masks texture and we go
here we can see that we can increase the
mask side or we can deep we can contract
it as well and we can also smooth out
here make it less smooth or make it
softer and this means that obviously
right now there's a blue background but
if I wanted to put anything behind it I
could if I go into the segmentation
background and if I select texture and
hit new image texture again this could
be an external texture it could be a
video but it's hosted online or it could
be an image that's hosted online online
or we can go into downloads and I had a
picture this is be fair this isn't gonna
wrap quickly so it just it doesn't wrap
properly but you can sort of see right
if you were to stretch to stretch this
out properly and map it to it it would
it would make it appear as if I was it
this is City fair and you said you can
do this videos too so you can do a video
so I mean yeah you can do it video as
well so host to see a City fair every
day yeah I wanted to right so for just
older for me that for a second there we
go
so you can see that there's this
interesting this is an interesting
another tool that can be used to do
really cool stuff again directly in
spoke out this introduced me to
something I actually wanted to talk
about which is like a powerful part of
sparking art which is the patch editor
we're not going to get too much into the
patch editor today but I want to talk
through some of the things that it that
it has some built in for free so if I
hit view and if I hit patch editor you
can see there's this kind of blank space
that opens up at the bottom of the
screen if I right click on that space I
get all these kind of interesting
options and the left-hand side and
basically what the patch edited allows
you to do is create triggers and create
different actions that can be applied to
any parameter pretty much that you saw
in the inspector so that means so if we
look down here it has loads of different
what they call kind of like nodes that
you can manipulate so faith landmarks so
I've got cheek chin herbal eye brow
interaction we've got like a blink here
you've got eyebrows lowered eyebrows
raised head nods head rotation so if we
pick head nod for a second so it creates
this kind of like node here it takes in
a face and it outputs a sort of
treatment impulse is has the head being
nodded so we need to input the face
actually into his head not tall and the
way to do that is by typing in face and
here the first sort of thing is face
finder which finds all the phase in the
current camera frame so we want to find
a face and right now there's a one count
if we sort of see that there's one face
in the scene and if I hit face select I
want to select that face index area and
put that face into the head nod so this
is kind of like a visual programming
environments is a visual programming
environment yeah and this is kind of
like what cs50 G with scratch and also
what some other services like unreal and
unity with their third-party plugins
sort of offer
yeah but why I like this is it so easy I
mean the difficulty becomes if you stop
doing more complex stuff actually this
the nodes visual base isn't the best I'm
but you can't avoid in JavaScript and
bridge it directly into like actually
important javascript and turn that into
into the this engine into this kind of
patch editor yeah that's pretty cool
into into custom kind of notes and
people have done that for some really
good people in this pocket community of
crazy it's not really cool different
solutions to problems like 3d collisions
and stuff like that it'd be cool to
maybe showcase a few interesting AR
projects - maybe yeah maybe towards the
end yeah we can definitely check out the
spoke I'll page see some people have so
had some questions of above sure Andre
was asking if there was no need for a
stereoscopic camera because the face
orientation shadows yeah so actually I'm
not quite sure how the tracking works I
had something about this but it does not
require a stereo at all it uses just a
single camera like right now this webcam
is and two cameras it's just that
there's no depth sensor anything I think
it actually don't know how that works
Adam inner shadow shadows would make
sense but that would kind of well no
that would make sense because I was
completing the light that it actually
provides in the software versus the
light that it's getting from the
environment somebody earlier asked a
question about whether it's possible to
import ambient light do you know if
that's possible to like take light from
the real world and like make that appear
in spark AR yeah well there is something
like ways to get around that you can you
can some 'pl areas of a scene in in from
the camera texture so the camera has its
own texture here under texture
extraction and then you can use that
texture like in you can use the alpha or
RGB values of that texture in turn as a
sort of reference for something so yeah
you can take the UM be away from the
scene okay it's enough way that's how I
would do it I would take the camera
texture data and I know it use that to
manipulate the parameters of the asset I
would like to that make sense yeah
Natasha was saying living this live
stream living a lot for you guys AR is
fantastic brilliant m4n Joo is the same
that this is used to blur backgrounds
and a newly released Microsoft team's
video conference I don't know if this is
this spot Koz is visiting me but
definitely probably something similar to
that and I know that like Android also
on
camera app have like an interesting like
background blur feature and stuff like
that this is definitely becoming into
sort of the vocabulary of visual the
visual kind of images were seeing on
like people are loaded online and stuff
but this is why you can manipulate it
actually you can be append the
parameters of it and not just have
something automatically sort of a
certain blurriness level of stuff like
and this is what this is actually sort
of a point actually it's good to dwell
on it to you for a second and why I
really like AR and is that stop thinking
beyond just like a background blur like
stance or thinking about how you can
make that dynamic in some way and maybe
this is a good point to bring an example
from from the Spock AR from this pocket
sure do you want me to cut the no it's
good I can I can just wait it up here
okay you have an actual project yeah so
this is a really cool like example of a
project that's a Spock I argue blows of
sort of tutorial tutorials of like how
to use the products it's really really
cool and there's some really good sort
of they go through more advanced stuff
in the tutorials like the 3d objects
don't just have to be static on the face
they can sort of wobble and rotate and
you can clog anime different Mike
explains and stuff in a book but what
I'm going to go too much into that today
but one thing I wanted to show you here
is this is a really good example of like
the node of the patch editor being used
so we have a scene and in the scene we
have a sort of rectangle which has this
kind of overlay of like of a diffuse
texture here so if we look in the
rectangle overlay under the material we
can see there's a friend color map and
if we click on that material we can see
that the texture has this blue sort of
patches as created next to it and that
means that it's actually a patch in the
pouch editor here and you can see it's
also blue here now if we look into the
patch editor will see that it says
screen tap and there's a business all
the imports here and if it's tab is
increasing a counter and the counter
number is searching an option number
from these colors so this means that if
we tap the screen and we
simulate touch by pressing that and it's
going to change the color of the filter
of what we're looking through and this
is really cool but one way I would say
immediately that could kind of elevate
this potentially or provide a different
sort of more interactive or dynamic way
of just this kind of standard filter if
we were to insert say a face tracker
here so if I go into the canvas and I
insert a face tracker into the scene and
if I put this onto this second insert
apply this to this okay so maybe if we
make the face motion visible for a
second so right now you can see that the
same frame is being used but it's
actually now trapped to the face it's in
3d and it gives you this kind of really
interesting 3d and all the patch still
works like I can touch and I can change
the color and it's kind of really cool
um Savoie I can show you this in a bit
more dynamic way it's so much more
immersive yeah so here look now it's me
wearing this and I can tap and I can
change the color of my face this is to
say that like there's an incredible jump
in how how crazy is that jump between
like between between conover static
although this text is dynamic as well so
I guess it's there was some dynamic
stuff in it Londyn represent yeah London
shout-out you can tap and sort of
immediately change it and this is like
for essentially just two clicks this is
a really powerful feature idea yeah I
mean yeah the change in sort of how
interesting and dynamic that was is it
really speaks a lot into like small
details make it big yeah and like that
wasn't hard to do and I'm not something
I would suggest sort of thinking about
me developing high or applications like
okay so this is this is the effect I
want like a background blur but how can
I tie that to a movement or to something
in a sea
that can create this kind of like
dynamic aspect to the filter so really
capitalizing on AR as a platform yeah
I'm like an Spock I are particularly
like so there's so many like if we look
down this list here like there's so many
different things you can like blink
eyebrows load eyebrows raised you know
all paid shake head rotation mouth
object tap right koi closed
surprise face smile all of those could
trigger a different action in you know
or a change to the to what is occurring
on the screen and some of them can be
sort of signified you could say you know
bring up a text things saying smile to
activate or like smile and then maybe if
they smile the cadets flashes like a
camera or something like the creative if
you can really run what holding Spock
are and that's one of the coolest
reasons why I think the apps really
successful ya know it's tremendous like
they're just that little small yeah it's
kind of illustrates like how you can
take something and really take to the
next level with this and like if we if
we bring out the fact if we drag in the
face tracker into the hair like into
this into the patch editor you can see
what it's doing it's like finding a face
it's selecting the face and then it's
tracking the face and you can see
actually that you've got the 3d position
if I think before we play this for a
second
you've got the 3d position without using
x y&z here you've got the rotation value
and you've got the scale value here so
you can actually take any of those
parameters and set them so if I were
like when I'm this close to the screen
or this far away from the screen I can
change from sizing aspect or some color
depending on the position in space so
this is kind of where you start thinking
about the spatial storytelling behind
they are like it's very easy to just
attach a face tracker like well I've
done right now and create this kind of
cool kind of thing but like what if like
the color gradient change depending on
the position what if like we started
introducing like these kind of details
like by tracking them into space what if
I had to be in a particular rotational
position in order to activate the
feature you start to sort of see how the
play of the kind of experience becomes
part of creating really good
sort of user experiences and fun kind of
filters for people and right now it's
like seeing us like filters like to
share but you can if you step this out
into the world where we start developing
like narratives and stories based on
user interaction or like crank avatars
and cram is kind of digital memory to
yourself it's just like a really
powerful going forward it was a great
way I think it's like I'm thinking of
ways to tie it into like you know
somebody even commented on Thanos for
like bringing this into marketing for
movies where there are characters that
do things yeah you can so we can um yeah
if you if one second I can bring up some
example so for a view if I just quickly
loads I can go into this ba guy our
community great buddha was asking is
this the perspective parallel model and
I'm actually entirely sure with that the
perspective that was a perspective view
that we were looking at in the V I think
that's what the question yes or
signaling talk about parallel projection
onto a plane oh I see if I'm
interpreting this correctly I think that
would make sense I guess yeah can you
remove the Freneau says armed ray I
forget offhand what Fornell is -
actually it's been a it's been a little
while for now in 3d
oh I see the reflection relative to the
angle of view on a surface can you do
that I don't because there's a
directional light I'm assuming that
that's part of this lighting algorithm
definitely change once you've got the
track element in the scene you can
definitely change the lighting position
and that will change the relatively
change the the condition on the face
like there's an example in the spot
guitar tutorials where you've got a
flashlight and you're holding under your
face are you know holding it it's it's
on your face and they create a face
tracker and the light kind of actually
occludes off the mask and corrects is
kind of interesting look like the flash
like on a scary story look and that's I
guess something to do with it the
lighting we got a Arturo J rail in the
chat asking how are you bringing up that
search yeah that's a great question it's
a right-click in the patch editor so if
you right-click in that space and order
to access a patch alert if you hit the
patch editor that's how that's that I
did it that's how it will appear in the
bottom also shout-out to AJ make sure
it's repeating part of the chat a day
Adam what's up hello Adam I'm kissing
face triggers does that exist
hey guys pretty much therefore the duck
face selfies says Andre can you check
hands yes you can definitely you can I
didn't okay that's it sounds cool but
actually there's been some cool ones
doing it like someone did a baseball
once be like held your hand up and the
baseball like so you can kind of see the
potential there's also something I
wanted to mention that we didn't get to
which was the texture tracking as well
so in there we've done a lot of
front-facing camera but there's a really
big potential for in Sparky art
especially for more advertising and
marketing sort of minded people for the
rear facing camera to replace
advertisements and how that works is so
it detects a plane which is any sort of
surface but any kind of post or image or
video is a surface so if it knows what
that image is it can actually detect you
can save the Spock AR when you see this
image replace it with this
and it can be any scale in the world
because it's actually just digitally
replaced so you could take massive
billboard of like you know any sort of
big brand and you can completely replace
it with a video or your idea you can
make screens out of flat surfaces I
think it's really cool people could also
potentially replace other advertisements
maliciously with this right yeah and I
wanted to mention the Burger King
actually just released an ad campaign
where you can has loads of their
collected loads of the collected
competitors like McDonald's in different
burgers and different deals and if you
hold your phone up to their ads it sets
them on fire I like that and it brings
out the whopper in says you know what
book and you get a free whopper out of
it as well but it's funny because that's
just like showing how like it's actually
kind of nefarious uses something about
like of visual worlds like the world of
advertisements around this is can be
like individually tracked and tog did in
this interesting way like why in that
example with the postcard we saw that we
had interactive text on that actually it
was a dynamic text so depending on the
device of his being used it would show
that they the the current day and the
current sort of location but that was
tracked directly into the physical world
so you can imagine it if you were
tracking it sort of texture playing of
an advertisement that that if that was
pulling from something that was personal
but I you know there's some other data
they hadn't they hadn't heard over data
that was available
you could individually sort of target
advertisements as well why this you can
see that not being too far away yeah
yeah like the mobile reality film they
do saying how do you think advances in a
RDR MRX I will change storytelling
nightly I usually um what I think that
we're moving this is great art school
that came out onward there talks about
the digital sort of me reverse where
essentially if you think about not to
get too deep into augmented reality sort
of fear all ideas but if you think about
wearing a pair of IR glasses on your
face and anything you look at being able
to be sort of captured
to act and create a 3d model of
essentially in real time oh let me just
saw here it's creating a 3d model of my
face in real time then conceivably if
you walked around the world it could
generate a map of where you walked
around and then you could basically
create a mirror verse to the actual real
world that's one-to-one a perfect map of
the real world and there's constantly
updating voice the people are wearing
headsets that's getting bit far down the
line but essentially that's to say that
imagine like an alternative sort of
parallel universe that's digitally
generated and created that can be
manipulated in all the ways that vegetal
gives us access to we can pause time we
can rewind time it exists eternally
something it's pretty crazy if you
something about possibilities like that
you could walk through any place that
you've ever visited you could ReWalk
that path right appear to be there
you could go to places that you're not
currently packed by going into a model
of that place it's being created in real
time from other people obviously want to
make obviously find the road because of
the computational power for that would
be pretty insane but the idea is
theoretically definitely achievable yeah
no it is
and that's there thank you a favor put
that link there that's the link to the
young to the mirror while these what I'm
kind of thinking about that's a lot want
me to give up recently but more simply I
mean sort of a more proximate change I
guess in storytelling will be the way we
sort of take photos and interact with
each other the visual language of our of
our photos online and of how we share
and how we share experiences will start
be sort of augmented by like you know
people like the hotdog filter on
snapchat or things like that these are
like small examples of what potentially
could be a big tear in the digital
physical divide which is now facilitated
by this kind of portal phone but could
be any kind of visual way of altering
vision right cool cool all right
um let's go ahead and are you ready to
switch to your computer we can take a
look at some um yeah I just wanted to
like highlight the chat oh yeah take it
however much time you want yeah I'm just
gonna quickly figure out how to
how to data Steve EoE these things
Stanford is better what a burn
Stanford's didn't really good stuff
right now when will you livestream about
STL to library via C++ we don't have any
hard set plans on any of that I would
have to take a look at it I think I've
done STL what c++ in the past but it's
been a long time Adam says what's up
colton not much - doing some AR we're
here with Connor Doyle shows - Connor
for his awesome spark a our sort of
overview today it's been yeah really
cool platform I'm actually really
excited about the possibilities here I'm
trying to load up some of the examples
so you can cut to see how like Smart's
it's on Facebook so you know naturally
there's a lot of obviously we want to
make sure we're not sharing private
stuff I think exactly what I'm drawing
out to do right now and such a hacker
man look at you I know I'm trying to get
out of it
it's getting really uh I don't know why
that's and Adam is saying what's up
Connor what is up Connor what's up not a
lot right now
meet me try to figure out how to remove
um a whole bar from from this and then
AJ saying hyper a leaf can you uh are
you able to right-click like a
particularly and maybe oh yeah I need a
moat or something yeah that'll be a good
idea I'll do that voila he's in ICU
right side or so sleepy I'm not sleepy I
might just look tired actually yesterday
I was not feeling well at all so I slept
much of the day yesterday feeling better
today but that might be why I look
sleepy I'm just gonna pick one I think
there's a loads here this is an absolute
turn of demonstration video it's crazy
because people developed whole like I
said people is about it whole UI is in
this so like you can you can bring in
textures and assets and like animate
them onto the world so people have like
you know the instructions for how you
know we've we've looked at kind of like
kind of funny example as a guest today
but like three instructions of how to
like potentially deal with paperwork or
like access certain devices you can you
know you just track the track the image
of like you know
how to assemble furniture for example
and if you hop it over and then lift it
up it would basically you could animate
the thing building itself and you could
just create and in AI you can do it this
could be I mean the potential is crazy
let me just yeah let me just load up any
one of these real quick Adam was saying
is a I used it all for self-driving cars
and testing them with it in self-driving
cars I think it there's a different type
of tracking in a our cars so like Google
Wave oh for example uses has a light on
and sensor on the top of it that's
creating a depth map that's then sort of
put into sort of these machine learning
algorithms to basically there's
different ways of tracking the world
around it Google Wave owes doing I think
with lidar which is kind of like a more
advanced version of tracking down this
you could say it's kind of like using
the tracking of augmented reality in a
similar way but I wouldn't say it is
augmented reality and that the sense in
the sense that it is overlaying physical
assets into the world for our visual
that lets me use more for like detecting
physical things in the real world and
adopting an algorithm yeah it's like the
car the cop it's not it's not off since
it's being changed as a car sense so
it's like a tracking for the car or
tracking worth it's kind of kind of but
that's an interesting question actually
to think about like what about augmented
reality for non for non-sensory for
non-empirical use kind of an interesting
question um in for a completely off
topic I'm super frustrated the
limitations bootstrapped if people use
other packages like semantic UI and
designed a compliment bootstrap probably
I think it's I mean it's up to you as a
developer but Samantha you guys really
cool I've been diving into a little bit
I think it's definitely worth checking
out okay I think I'm just gonna click on
will just load he's home really funny
oh I Kenny phones pretty cool it's
pretty cool how do I open this so for
now so if you just click on the if you
just click on the actual page itself or
right-click and go to like view page or
whatever yeah see there now that your
chats not
okay so I see that you can take a look
at that I'm gonna swish your laptop okay
cool yeah that's good all right
so you can see here like an example of
that someone's created with this
morphing animating it's Christian
venable shower Christian is great so
they have like they have like a
background potentially that backgrounds
a 3d model as well but they have
minimally some models rotating around
their head and also one sort of
juxtaposed on their face or insert it
onto their face so you can see here
using a face tracker to track that
morphing sort of crystal on the face of
the using a screen tap to change the
color oh okay that's cool it looks like
and they're using segmentation so it's a
very flexible format you want to pour me
another example of being sure yeah I'll
just look like this one this looks like
a similar one I'm guessing well it looks
really a realistic apart from the guy
referring to the guy not looking
realistic or what's up what was on his
face would you want AR to have a part in
theater or would you rather keep theater
as it is I think what he did that's a
great question I'm really interested in
AR in theater um I think it's a big
there's a big space there for spatial
storytelling in spatial experience to be
kind of infiltrated by this kind of
digital tech and there's been examples
like The Tempest which is a Shakespeare
play and the RSC Sarah Ellis
I wasn't Tara Harris that no it wasn't
her at least my apologies but um The
Tempest was done with an incorporation
with Intel and they basically tracked an
actor live and then projected into this
kind of like surface the kind of digital
asset so it's like a reactor was on
stage performing but it was being
tracked into this kind of magical fairy
on stage so that's like a early stage
use of AR one of the difficulties of AR
in theater is that like right now the
viewing device for AR is like glasses IR
glasses all of the phone and the glasses
are very expensive and kind of a
cumbersome currently and the phone in
theatres those are always a worry that
people are going to be like taking video
texting and they're going to be
distracted from the live experience so
there's a boundary between like wanting
to add digital assets into this
and then also wanting to preserve the
integrity of the of the live experience
and that's something that's been always
a challenge in my live music for example
as well or comedy and in some comedy
events people actually take away the
phones now and give you like a seal the
bag and give you this kind of bag but
you hold see you can't actually access
your at the event that's funny
so there's kind of interesting questions
when I get to that but ya know I
personally think that I always
definitely coming into the spacial
storytelling realm is definitely going
to be an interface with theater if
that's going to replace theatre I don't
think it's a question of ever replacing
like vo it's a V I'm gonna replace
cinema is we all gonna replace movies is
a are gonna replace you know immersive
experience no it's just a different
platform for viewing myself experience I
think it's a completely different sort
of platform rather than a sort of
replace me in that sense bit more more
closer to installation or closer to like
smaller experiences and there are really
cool theater companies like punch-drunk
which developed by Sleep No More and
things like that they're exploring these
kind of immersive and digital sort would
divide that's a great question and
something I'm think you know a lot I
feel like for especially for like
theater like the phone that's a great
point actually sorry I think it's
unplugged in theater in real time would
be awesome that is a really good point I
think that would be really cool as well
but the question would be no again and
people have done actually done some time
always in a are the question is like if
I've got my phone out and I'm watching
the theatre through my phone when I can
be what it what can I just use like a
projector screen with the subtitles on
but why what would be the additive value
of I are tracking the text into the
space unless it was part of the
experience and that's a question that's
another way of thinking about like AR
and VR a lot of people will have ideas
about AR that could be achieved most in
more seamless ways by using just video
and that's where it becomes kind of
difficult to think about but I would
agree I think the supplanting theater
would in real time would be amazing to
track it into AR but only it was like
integrated into the experience otherwise
it's just why wouldn't it just be sort
of like a projector screen yeah these
are kind of things to think about that's
true yeah having like an animation of it
like going around people like the
quotation thing that you
surely yeah like if there was a some
sort of like if the play was if the play
was although the event was very
interested in words or if there was some
kind of other real-time aspect maybe the
text was being like inputted by a
certain but it was tracking with the
heart rate of people in the room or it
was like maybe it was there was some
other thing that was inputting into it
well play was concerned about text in
space there maybe that's something to
think about but otherwise it yeah
Jacqueline had with Sherlock like
thought bubbles and be cool Bobby cool
yeah yeah no tracking tracking tanks on
to people in the space would be cool
oh yeah like a speech bubble yeah that
would be really yeah the text what
appearing on them would be cool but
again it would be like you're trading
off the fact of me seeing in stereo to
be seeing the whole experience even is
still like flat mono video on my phone
you know in a smaller frame so it's a
question to think about opera also is
something I'm actually talking to
someone for comedies about doing so yeah
I'll probably incredible again it's the
question of think about the audience to
go to theaters like our audience is um
people that aren't maybe on is tech
savvy as us potentially and how are they
gonna and what if they encounter
difficulties wife they haven't charged
fo what if they don't bring a phone do
we have to buy phones for 2000 people
how do we charge them how do you make
sure no one steals them they like these
logistical questions that come in to
like any kind of a most of experience
event for this I sort of feel like like
a headset makes more sense do you agree
or disagree I disagree the reason why I
disagree is because if how I our
headsets are currently work is that they
track the space by having a clear line
of sight to the space and they they kind
of use a depth sensor and image to
generate it but in a theater you're
sitting behind people and there's mobile
people so it would might interfere with
the track but I guess if and then you
start to go well maybe there's a way to
figure out you know maybe for a limited
audience definitely I think I our
headset would be good this would be for
a limited life yeah
yeah yeah hey our glasses like 3d
glasses never seen never used an AR
headset okay the hololens pretty cool
and that's the only one that i've used
to be honest yeah Holland is amazing and
the hololens just an apt hololens -
which is the next iteration of a
hololens a magic leaf is also a big AR
headset essentially they're a pair of
seafood glasses the position on your
face with some sort of a computer behind
and sensors in the frame that track for
space understand it and then projects
Euler directly in Torretta neuro onto
the piece of glass objects that then
appear to physically exist in the space
but actually they're just being
projected onto what you're looking
through if that kind of makes sense
right yeah
glass Enterprise Edition is that a new
thing coming up Holland's oh that's why
the Google glass Enterprise AJ or like
hololens are doing a kind of a much more
enterprise focus sort of leanings for
training so like imagine if I would
wanted to train someone on how to use
equipment but from a specialist but I
didn't want to fly the specialist out to
X location or I could have someone
wearing a headset and train them using
it in the actual physical space but not
have to have a facilitator there to just
show them how to do it
do you know Adam is asking is it a are
safe for your eyes and I'm assuming he's
referring more to the hololens I all
safely always I'm unaware of the I'm
unaware of the literature on that I'm
sure there is some research into it but
we're still early days so there's been
no researching to prolong things maybe
two and we'll show you one more example
of like of something that I've done with
only one of my collaborate collaborators
Andrew Daffy oh shut up - Daffy London
if you check that out he's a really
really cool artist there's a based in
London as well he's based in London yeah
that feel of this if we so basically I
wanna go there was this big phenomenon
on Instagram about an egg
oh this yeah you showed me this I
realized
fan of this so there was a big phenomena
about this egg so what what I decided to
do with Daffy was basically track this
3d model of this egg to a face so if you
see my face like is Tony I'm actually
controlling the egg and I'm not in the
name of this project a blue egg meant in
reality mode up here and that's Daffy
lengthen Donna you can follow him on
Instagram we can follow me on instagram
and i am connor Doyle but here so if we
look at the scene there's a lot going on
here but pretty simply there's a face
tracker that you can see here that's
controlling the rotation of the egg and
if you're close enough rotor changes to
you yes I'm like we did this fade so as
I go close you can sort of see it
becomes my face inside the egg so this
is just an example of like a cool way to
think about Spock they are in like not
just oh it has to be their face or it
has to be something on the face or it
has to come like this is just an egg in
the space it's been controlled by the
rotation of a face you see how it's like
a different way of thinking about that's
funny yeah egg on my face it's a
different way there's so many puns on it
there's different ways of thinking about
crank a are experiences that aren't just
oh let's put some glasses on someone oh
that's like retinas although those are
really called as well um the same idea
over and over again is gonna get tiring
yeah like to think about oh so this is
playful people try this intermediate
like the world it's also very clean and
simple I'm completely lost by the egg
thing yes sorry so just to put it in
context um someone's loaded image of an
egg on Instagram which is a social
network and I think it got the most
likes ever if you just google Instagram
tabloids or something I think it got
this thing in a just a regular egg is
instagrams most like most ever 35
million people have liked it cuz League
winners birth
I was like 255 million likes of an egg
that's a TL for me I did not know that
yeah so we created the crate this sort
of air gets this sort of as a fun thing
for it and if we look in the patch
editor here doesn't say bring the
pattern iturra because they can see that
there's a lot kind of going on here but
what we the the main challenge with
developing this was I'm not having it
track directly to the face so if it just
tracks to the face then if I were to
move up and down it wouldn't stay on the
floor so we had to take the value of the
floor and a lot of that into place and
we also wanted to do some smoothing on
the head rotation so as you can see on
in this kind of main perspective view if
I rotate my head left and right it moves
a lot faster than the egg moves so we
wanted to quite a kind of latency on it
so it sort of moved a bit slower there's
a little bit more organic more more
poetically more poetic motion whether it
be so yeah this is although this is
doing its unpacking the 3d rotation of
the egg and then it's applying some
smoothing and dampening here and then
we're doing some other math to control
the rotation of it and yeah and then
there's a couple of other cool things we
can shout out here that Oh
if we look actually under the egg it's
it as is kind of like reflection that's
actually just the egg absorber another
version of The Hague upside down that's
creating this cool reflection to it
and then this on front of the camera
we've create this little vignette so you
can see that in the main just like this
up in the main thing that's kind of like
a swiping yet that comes in here so yeah
so that's and we did some lights here
these are dynamic lights are lighting
the egg alejo asses on your Instagram
page nobody's on augmented reality you
can check that out egg meant it on this
core reality did you make it as appears
we made it to page exploded I love the
rotation out of this part of it you see
how that's very taking the egg yeah yeah
there's incredible texture in with up is
it's a very well balanced egg it's well
people wanting to crack the egg we had a
lot of requests for let's crack the egg
oh yeah very cool actually um you'd have
to get an atom you have to get another
3d model to represent the crack
yeah we didn't we didn't have time to do
the crack yeah that would be nice vitu
though yeah so this is just an example
of where we can go with this and yeah in
a bit more advanced way cool yeah that's
a really cool project very clean and a
very good I think illustration of like
taking a R and not doing the simple
thing you've seen over and over again
which is somebody's face like having
something attached to it that is cold I
love to say that there aren't cool
things like there are so many examples
of really inventive ways of playing with
the face and the image but there are
other things as well that can be thought
about this has anything potential to
really get people's attention in a brand
new way yeah I really sink their teeth
into yeah introducing like as I said
like when you're thinking about
designing our experience is introducing
some interactivity in anyway a sense of
play and a sense of kind of
understanding of that we have a kind of
panic space to play with not just a kind
of flat plane then that really will
introduce you into understanding
immersive sort of storytelling an
immersive in augmented reality yeah it's
exciting and this this you know again
this service spark they are completely
free you can get it you can download on
your computer you can make stuff and put
it on Facebook and Instagram yeah
and it's tools like this that will pave
the road for Elena more people getting
involved and you know the more people
get involved the more people compete and
competition brings about cool things and
the communities growing in the
Facebook's walk out community page is a
really active community and it's so
there's you know people encounter
questions they're really quick to answer
there some incredible creators in these
space but really kind of pushing the
boundaries of what we see on our phones
so I think this is a really exciting
step into augmented reality water and
one sort of tool in a set of tools to
explore storytelling in this space yeah
I'm very excited to see what the next
five years brings us cuz it was less
than five years ago that this stuff all
sort of started coming really to the
forefront yeah modern technology
exciting yeah well thanks Connor for
community space jam really appreciate it
thanks everyone this is an awesome this
is an awesome introduction so today is
Friday tune in next week Nick Wong will
be joining us on Tuesday for part two of
our CTF and then Friday we'll take a
look at the jQuery JavaScript library
which some folks in here might be
familiar with the next sort of extension
are sort of web development threat so
people are kindly saying
thanks for the awesome stream as if
everyone burg m14 jus so yeah have a
great weekend and join us next week
Tuesday 3:30 p.m. for our CTF and then
Friday with me at 1:00 p.m. for jQuery
this was spark era thanks again Connor
for this awesome stream enjoy your
weekend everybody bye bye bye bye