(music)
>> Sponsored stream.
What is it like to do a video feed thing
with Azure Machine Learning stuff?
So Cognitive Services, APIs for AI developers
or regular developers too.
This is meant to be accessible to everyone.
You don't have to be a Machine Learning expert for it.
We are gonna be doing it on video.
'Cause I wanna eventually be able to
get like the emotion from a video feed here
and maybe like color the panel or something different.
I can start turning red when I'm getting trolled.
We're gonna be doing computer vision today.
Like parsing a video, have it analyzed some stuff.
We may end up having to mix a few of these things together.
We wanna go log into the portal then, right?
We got this right here for face recognition.
Completely brand new to this,
we're seeing what we do as a complete beginner.
We're gonna need a face client or an SDK.
And maybe this is how we end up wanting to do it.
We analyze the image on just like a poll,
like once a second.
We're gonna wanna do these API calls,
I'll probably wanna use the Python.
Let's create a folder for this.
So this is video-mood-detector.
We are gonna want an envfile
and inside of that, we're gonna say Azure face key..
There are two keys and there's an Endpoint.
So We'll call it key one, Azure face key 2.
There's the Endpoint,
and that ends up just having the project name that I have
cmGriffingfacerecognition.cognitiveservices.azure.com.
They don't show the key, which is nice unless you make 'em.
We got our Endpoint here and we got the location.
And this is in photos specifically.
So yeah, the way we wanted to get like the mood
from a video feed,
that would probably involve sampling
just an image from the feed on like an interval.
Looks like we can use JavaScript for it.
Oh, and we can try the face API in our browsers,
we'll check that out too.
Here's our quick start into detecting faces
using the REST API and JS.
In the meantime, you can get an npm install them.
Probably wanna do an npm in it.
All right, so, we got that add.
So the face API, is what we're using.
Ah, perceived emotion.
If we go grab this one, what mood do they detect?
Contempt, sadness and somewhat neutral, all right.
One face detected, try one more.
(laughing)
I don't know what that one is.
That one is surprised.
Yeah, sounds about right.
That is mostly surprised.
So this is the main part, perceived emotion recognition.
We can identify the facial hair, fancy.
See if they'll actually identify this
as a mustache, press x to doubt.
We'll see, We'll see.
We could do it from like a video feed.
What I'm thinking here is,
to get like a live analysis of the video feed
down in this corner here.
We'll end up pulling the video
sending that up on maybe like a one-second interval.
And I think that accomplishes what we're looking for.
Their API even said up to 10 per second.
I don't think we need that much fidelity.
You could even go maybe like two per second
and it's still worth.
This looks pretty straightforward too.
It's just a post of the image.
We have our params in here.
We just make a post request.
So, we need to install some libraries.
We're gonna do axios.
We can start out just sending one of the images we have,
like here on our desktop.
So we'll do that.
We're gonna create our script.
We'll call it test.JS
So we need to bring in axios.
We also need file system stuff.
So we'll just do fs.
We need an image.
Image data equals fs.readfilesync, oof.png for now.
Gives us back a buffer.
This giant URL.
It's a post.
So response is equal to axios.post.
The Endpoint is something we have Nadar env file.
So we're gonna have to bring that in.
Let's see if I can remember what I named these teams.
How do I reach these keys?
So Azure Endpoint.
So image data, that might be okay.
Just sending the raw buffer.
I don't think so. We'll find out.
This is where we set that key application.
So the key here ends up being, Oh man.
Azure Subscription key.
I really forgot what I have it as.
Face key, okay.
So let's start looking at this more.
So return face attributes is what
we're looking for specifically.
Ooh, how old do they think I am?
This is gonna be, I want all of them.
I want everything, right?
At least for these first couple.
We'll do, yeah, all of it.
Every single one.
So return face landmarks equals true.
Return face attributes ends up being this string.
So here's something interesting.
Default value is 01,
but they're saying three is recommended.
Recognition model three.
I think detection model is
like finding the faces in the thing.
Recognition is like identifying the attributes.
I don't think we care too much
about the return recognition model or detection model.
So we're gonna remove those for now.
So we have our url crafted now.
All right. So, from here we just do response data.
We should probably do status as well.
We can see the face API in action
and this is basically what we're gonna be calling out to.
If we look at that oof,
we see that is just almost pure happiness
with a little bit of surprise.
He's a surprised happy..
So yeah, you call require.env.config.
That call right there goes out for that .env file.
And then it brings it in onto processed.env.
So let's try one more image just to show it off neutral
a little bit of contempt and happy.
He's happy, neutral, contempty.
So let's look at the data we're getting back.
We sent oof.png
They gave us like a head pose.
I'm smiling in it.
A 100% smiling basically.
Just as a reference,
this is the image we're talking about.
And now it's in your nightmares.
Face landmarks, they even tell us where like the pupil
and nose and all that stuff is.
So landmarks is a cool one.
When we get to attributes,
we see head pose here and approximate gender.
They think I'm 41.
I look so young.
It's gotta be the wrinkles, right?
So let's try the pod champ.
So once again, the oof, 95% happy.
Little bit of surprise in there.
And then none of the others.
They are noticing that I'm wearing a beanie.
A 100% confidence in that one.
Forehead is occluded.
Oh, fancy.
But man, yeah, we're already there.
We're getting responses.
We actually made it farther than I thought,
quicker than I thought for this part.
The way to speed it up, is use less of the recognition.
For the most part, we are requesting too much data.
Do we care about face landmarks? Nope.
We don't necessarily care about the age,
gender, head pose, smile, facial hair or glasses even.
All we care about for this is, is emotion.
So we're using recognition model three
'cause it's more accurate
But recognition model one might be faster.
So first let's get a baseline.
It's pretty quick.
Some of that time could even just
be like the script spinning up.
We will just do performance.now
and do our own subtraction,
Time to run, end minus start.
So we're looking at 835, I believe milliseconds,
which means polling on like a one-second interval
makes a lot of sense.
Let's try one of the other recognition models.
Recognition one, we see it improved
by about a hundred milliseconds.
We see one, two, three, four, five, six,
seven, eight different emotions.
That's what we would use to maybe
determine the color to display.
That would end up being part of maybe
the last step we need.
So what we need to do is somehow figure out how
to extract the current frame from my video.
Once we can get there,
we can save the output to some Json file,
and then have some OBS overlay.
Query that Json file on a timer.
Maybe this is the part we hold off on them.
So we're gonna have a server.
Which means we need to install express
And, you know what?
Let's go get some express JS boiler plate.
Let's just grab the Hello World!, app.get.
We do some async.
We need some fs-extra.
We use fs-extra to get promise based file system stuff.
So then here, what we end up doing is, const emotion data.
That will be equal to awaitfs.readFile.
And that file will be here
and we'll just call it emotion.Json.
When we spin up this server,
we also simply wanna do something
like this setInterval
and inside of it, yeah, we want this happening
like every second.
And this ends up being our test upload functionality.
We're also gonna have an images array.
We got hmm.png, oof, pogchamp, sure, suspicious,
and we'll do ugh.
Here, we simply grab image data as this.
This is images, images.length.
Start at one, (inaudible) by that.
As we get the image data here,
that's where our little upload script ends up going.
So we're gonna get that response.
The image data is this thing
we're dynamically loading in now.
We're parsing our key, get the response back
and then we can do fs.writeFile
and we'll just do it to a emotion.Json.
This is a totally different thing.
This is just gonna be overlay.
And then here we just simply wanna have
maybe overlay.html
And in here we wanna do body background transparent.
So then in here, we can just do setInterval.
This could end up having some lag,
which we don't care too much about.
We'll just say 500.
While we're doing this interval,
we're gonna do axios.get,
local host 3000 emotion.
Here, we almost need a way of determining what color to get.
What do we wanna call this, colors?
And we actually want this to be an object
in which the keys are these values.
We could just copy these, now that I think about it.
So anger is gonna be red.
For contempt, green.
Maybe like more like an orange.
Disgust, purple? Sure.
Fear, maybe just a gray.
Happiness we'll make green.
Neutral should probably be transparent.
And surprise we'll make like a yellow.
I think it works.
All right. So we got those colors,
when we come back here with a response
and we'll set it to neutral by default.
So now for each of these, we have a key
and we can loop over this highestValue
equals emotionData.neutral.
If emotionData is greater than the highestValue,
we set the highestValue.
We loop through these,
we grab the highest one.
This isn't gonna account
for like when two emotions are exactly equal.
This is just a quick, hey, is it even working type script?
We can use the emotion we extracted,
grab the color for it.
We do style, background color
and we set it equal too, yeah.
All right. So moment of truth.
Hey, there we go.
So it seems like most of my images
end up being transparent, happy.
And then bam yeah.
We have the source over here in the overload.
And ideally if we were showing the image
that would be good too.
And keep in mind, this is still just kinda hacking at it
because ideally this would be overlaid over a video feed
that we're extracting the image from.
So the way we're hacking the image
into the response here, not ideal,
but it's at least to like show us the,
image it's seeing,
what color it's overlaying over it.
We're gonna call it just images for now.
With that there, we can serve up the image
and we can put that onto the response data here
and in our overlay
we're gonna end up putting an image in here.
Class equals mock image.
Class equals wrapper.
Mock image goes inside of it, padding zero margins,
zero, get that nice and reset.
Got mock image in there.
Wrapper is then position relative.
And we're also gonna have a thing called image color.
You set that to background transparent
and we wanna make this have position absolute top0,
right0, bottom0 and left0.
Images and this is where the value from
emotionData comes back.
So emotionData.image.
Let's go double check it over here now.
Hey, and there we go.
Now we're getting some, some of it.
The object fit doesn't seem
to be taking as much effect as it should,
but bam, that's pretty good.
So what we're seeing with transparent here,
is that transparent just makes it a neutral.
It makes it just transparent.
So yeah, that's cool.
Let's go look at some of our color data though,
and where we see transparent,
it is more neutral than something else.
Maybe instead we just, we don't do neutral.
I think that one's happy though.
Questionable.
I might've messed up the colors.
That's the only thing I can think of.
Those look right to me.
So, the way we're doing some of this is off, right?
The highest value detection, but yeah, not bad.
So what we're doing,
let's go kinda look through the code
and see what we're doing in our server.
The important part is,
we're pretending that we're parsing a video feed.
Instead, we're just doing a loop
over a set of predefined images.
Eventually that might just be a quick
and easy replacement of just parsing the current image.
When we get that current image,
we read that.
This is maybe turning it into actual image data.
And then we pass that through to this API for the Azure end.
And we've seen that Endpoint.
It is a fairly simple, we're not logging it.
But it comes back with just some data.
So it says where the face is within the image.
And it says what attributes it has on it.
And here we're selecting which attributes we care about.
The more you add the longer it takes to run.
How much are these gonna cost? 300 requests.
Average latency they're saying 310.
So I've got some, some of my own like network latency going.
So it's an estimated 40 cents for a thousand calls.
Okay. We're totally good.
That's not bad.
Here's the thing we wanna do.
Why is anger not working?
As we loop over these things, what's going on?
It's probably something so simple.
I'm gonna be exhibiting the anger
dealing with this anger bug. Right?
We see happiness.
Oh, we see neutral even after anger.
That would be part of it.
Ah, there we go.
So now we're getting sadness, happiness, anger.
All right.
Hey, all right, cool.
Let's get the overlay showing as well.
So imagine, that's my webcam for sampling it once a second
and the overlay is just something where it might be
like the border of the image.
All right. We could eventually turn this
into like an OBS plugin.
The user enters their like Azure credentials.
Then they're on the hook for the cost.
That can be really fun.
We had a lofty set of goals.
This part, done.
The part that we're not doing is extracting the image
from a video
and then after video, use live feed.
None of this part is done.
Hardcoded video first then this second.
Man, I am a really happy
with how quickly some of this stuff came together.
Let's do a final like, hey, here's what we did today.
Here's what we didn't quite do today.
We weren't able to get this.
But we were able to test with static images
upload them to Azure and get back some emotion data from it.
And we see that process happening right now.
We're imagining that we're just sampling a video feed
at one second intervals.
We're uploading and getting a response back.
And from there our video overlay
would then have a color applied to it.
So here we see with some of these images as they come back,
it's either mainly happy, surprised,
there's anger, there's sadness.
We didn't have one that's mainly disgust
or contempt though, or fear.
We get these values back from sending an image.
We weren't able to extract image from a video.
And admittedly, I wasn't even sure how to do that.
And I'm still not sure how to do that.
But we wanna use FFmpeg
which is just a library for parsing and manipulating video.
This part we may look into at another point
just to get the thing working here on stream.
because we couldn't figure this part out though,
we wanted to make sure we're still like getting stuff done.
So we wanted to get an overlay in OBS going
where it's just like maybe a square
of the color we'd be doing.
We're using our images inside of there
to show you like which one got parsed into which color.
Because it's not a live feed,
we're doing it with just static images
looping through them as if I was making faces.
So if we go and see it in effect,
this is it just right there in the browser
for debugging and testing.
We created a very minimal server
for representing and holding some of this data
and serving it up to the browser overlay.
We can then look inside of OBS and bam.
We have this browser source
that we would then overlay over an actual camera instead
of these hard coded images and Wallah.
So let's actually look at what we did to make that server
and make some of that plumbing actually work together.
So we're doing standard express app.
We've got our app, our port.
We got some Azure credentials such as the Endpoint
we're hitting that relates to our specific service
within Azure and our key, which we don't wanna show you.
So we have it inside of our environment variables.
Some of the core stuff, right?
We're using .env to hide away the environment variables.
We're using express, like I mentioned.
And we're using axios to post our image data
to the Azure API.
And then fs-extra we used
'cause we wanted our file system stuff to be promised based.
Regardless of which version of Node we're on.
I believe the latest version of Node,
actually does have a promise interface.
We're just not using it.
We use fs-extra for that.
Here we have, you know, our hard coded images
and here where we're doing this interval
that is also kind of like the mocking out.
If we were processing the feed
this is what it might be grabbing
as an image from that feed.
And we're just uploading that using axios here.
So Endpoint goes in
and we see that there's a bunch of stuff
we can pass to this.
We can go dig into a few more of like
the face attributes type stuff, right?
Right now we're just extracting emotion
but you can extract all sorts of data from it.
We got recognition three.
This is the model use to like parse this thing.
Theoretically recognition 03 is more accurate.
Whereas maybe recognition 01
we saw was maybe a hundred milliseconds faster.
Based upon what you're looking for, accuracy or speed,
you could pick one.
In our case the a hundred milliseconds
doesn't matter that much so
we're not too worried about it.
We're uploading the image data
and then our headers, we just simply say, yeah
we're uploading raw binary, you know, just octet stream
and then here goes our key as a head.
That's the simplest way to just try it out.
You can just pass your data to Azure
and get back the response that we see right here.
From there, a little bit of the plumbing is
we end up creating a get Endpoint within our local server.
This gives us the emotion data into our overlay.
And then here, overlay is just the html page
that is the overlay.
So the overlay is communicating with itself,
so we don't have to deal with cores.
Here we have the images that we're serving up.
That's basically just mock data.
You wouldn't actually be showing that
in the real thing of this.
The server is quite simple.
Here's some of our images that we're using to parse
and get some data from it.
Imagine, this was over a video feed
and we were pulling an image out of the video feed
in real time at like an interval of once per second.
We could detect what emotion I am showing in my video feed,
like right here
and then put an overlay over it as I get angry
or as I get happier or things like that.
This right here is a real world application of in my stream.
I wanna put a color over my video feed when I get angry.
You could totally do that. So yeah.
Hope you guys enjoyed that.
All the stuff we did is actually quite simple.