>> Hey, everyone. Thanks for coming out.
As you saw this talk we had quite a sit of set-up.
It should be fantastic or a flaming mess.
It will be entertaining. May name is rowan.
The three of us are program managers on the .Net team at microsoft.
We'll do a bit of a fun talk. There's been lots of talk about
Ai and ml, lots of breakout sessions with details. We thought we'd show fun ways to
Plug ai and ml together. The things we demo are probably
Not things you'll do in your day job but the idea is to show how
Easy it is to use the technologies around to do ai and
Ml with .Net. When we think about
Tractor-trailer ml space, there are two categories of offerings.
There's out of the box pre-train solutions so this would be
Things like azure degree service where you send in input and you
Get back output and don't train the model.
We have custom solutions. You're responsible for building
And consuming. This is where there is no out of
The box offering or if you domain doesn't work well with an
Existing offering maybe you have specific key words that won't
Work with something more general. For our talk we'll focus on two
Different technologies. We'll use cognitive services and
Then we'll use ml .Net.
I'll hand off to adidi. If you were in my theater
Session earlier today you will have seen the more detailed version.
When you start to see something you've already seen, it will be
A quicker overview and we'll move on to quicker overviews.
>> Awesome, for those of you who haven't made it to any of the
Other ml .Net sessions, it's an open source and cross-platform
Machine learning framework made with and for .Net.
So today i'll show you a scenario where i actually have a
Piece of damaged music so music with notes missing and i've
Trained a model ml .Net to be able to predict the missing
Notes and fill them in. So let me give you a sample of
The damaged music. Nope, wrong one.
Great. So you can see here that there's
Some obvious things missing in this music, where it's
Represented by the spaces hear. For those of you who are music
Connoisseurs you notice this isn't completely rendered
Correctly but you get the point here. So if i play this...
[ Music ] so as you can tell that didn't
Sound super great. There's obviously some stuff missing in here so let me jump
Into the code to show you how i'm actually going to fix
This. So here i have the json file
That represents the melody that i was just showing you and all
I've done is pars that fail into a bunch of notes that are
Represented by numbers so you can see here note 69, and those
Missing notes are represented by note equals 0 so we've filled
That in and hard coded that in hear for this melody in
Particular. So if we take a look at the
Dataset i used to actually train this model
... This is a first step you have to
Take when actually building your model so this is the data that i
Actually started with. This is basically a bunch of
Songs that i have, and in it is all the pitches in each
Individual song and the duration of all the pitches and a bunch
Of other information but not all this information was necessary
And not all of this was useful for being able to train my model
So i actually went ahead and did a transformation to be able to
Make it in a way that's consumable for my model.
So a quick music 101 for those of you who don't know a lot
About music. So one measure is kind of a
Standard in music, and it represents a set of notes.
So what i've done hear is transform the dataset into a way
That you're going to predict the note that's missing which is
Highlighted in pink based on the other notes surrounding it
In the measure. So this would be a classification problem.
You're looking at the context in the measure and figuring out
What's missing here. So let's swap back to my code.
And walk through my model. Thank you.
So since roland did a deep dive earlier i'll walk through this
Pretty quickly. But hopefully you get the idea
How i build this model so let's take a look at this train code here.
The first step i've done is to create a new pipeline so a
Pipeline is a really important concept in ml.Net.
Your pipeline allows you to load in your data, transform your
Data and train your model. This is the basis for starting
The whole model generation process so after creating this
New pipeline i've added in the data. I've transformed the data in a
Way that will be understandable and consumable for mylearning algorithm.
I've specified hey, this is a classification problem so i want
To use the specific learner that's to do with classification.
And then here i'm actually saying, pipeline.Train and this
Is taking that input and the output and training my model.
Oh, the font bigger, sure, yeah. I'll make it a little smaller so
You can see what's going on. Sure. Hopefully this is better.
I'll just show you that again really quickly.
So creating a new pipeline right here.
Transforming may -- my data
Here, adding a learner.
And then calling pipeline.Train here.
And then the last step here i've done is to actually write the
Model to a zip file that i can then take and drop into my other
Application that i was showing you so i can actually consume it
And use it to predict. On the bottom here, i've just
Actually created a test case, so i can tell if my model is
Working before i go ahead and drop it into my other app.
So this is going to run a prediction when i'm building the
Model and make sure that hey, everything is working correctly.
So let's go ahead and run this. And you can see that it went
Through a few steps and it actually predicted correctly.
It says: hey, for this measure, the note is a, and that's
Actually correct in the scenario that i provided it.
So now that i've trained this model, i can go and find it and
Drop it into my other project. So if i go here, open my files,
Go into my bin folder. You see music model.Zip has been
Created, so i'm just going to copy this.
Go into my other application. And paste it here.
And now it's been added as you can see down here.
So now that i've added my model in, i can actually go and
Consume the model in this app and use it to predict those
Missing notes that you saw before.
So let's go to musicrepair, and fill in some
Code. The first thing i'm going to add
In here is building the features out, so this is taking the
Dataset that i showed you, the song that i showed you that has
The missing notes, and just turning them into those measure
Contexts that i showed you before so that we can input this
Into the model so the data has to be in that same format that i
Showed you before. The second thing i'm going to do
Is actually load the model so i'm calling load model, and then
Model.Predict will take in that feature i just defined here, and
Make a prediction.
And then the last thing i'm going to do is convert that note
Back into a number, so if you saw that the notes all have
Letter names, but to actually play them, we're going to have
To convert them back into the numbers that i was showing you
In the json file in the beginning so i'm converting that
Back into a number in this last step. So i'm going to rebuild this
Project and then rerun it and we should see some actual good
Music now. And look at that, instead of
Spaces where you see the notes in red is actually where we
Filled in with predicted music so let me play this for you.
You can see it sounds a whole lot better than it did before.
[ Music ] >> all right so we've seen
Something with ml.Net. [ Applause ]
I thought we'd swap gears a little bit now and look at
Something using cognitive serves and the text analytics api.
So in our application we've implemented a few different
Pages. One of those pages is loads up
Some feedback that's been provided to the .Net team over time.
This was scraped from blog posts. And what we'd like to be able to
Do with this data is detect whether it's a positive or a
Neutral or a negative sentiment which is a perfect use case for
The text analytics api. As you can see the moment we
Have the bar at the bottom which is grayed out because we haven't
Implemented this functionality yet. So i'm going to swap back to our application.
The class that's responsible for loading the data is this
Feedback service, and in it i've installed the text analytics
Nouget package and created an instance of the api.
This gives me a nice .Net api over the text analytics api and
I parsed in things like the region and my subscription key
Into that api. Now, before the feedback is
Passed to the ui to display its run through or it comes from the
Get feedback and i have a place holder to implement sentiment analysis.
So the first thing we need to do is create some
Input. Here i'm taking my input and
Turning it into a batch of feedback that's going to be sent
Up to the text apples api so rather than having to do
Individual calls i can stack it all up and send all the feedback
At once. Then once i've got that i go
Ahead and make the api call. Just passing in the batch, this
Is a nice .Net api so it's an easy wrapper over the top and
With the results that come back, i'm going to look over them, i
Will find the piece of feedback based on the id and then i will
Set its score to be the score that came back from the text
Analytics api. We'll rerun the application now.
And we'll see we're now getting sentiment coming back.
And we've got a mix of feedback here, some positive, neutral,
And negative. We thought after the music demo
That this one seemed a little bland, so we have edwin na on
Stage that we've hooked up to some lights in our application
We implemented a light controller using a serial port
To communicate with the edwino and it's sending through
Commands to set the color of the lights depending on the
Sentiment so swapping back to our api, we can get some nice
Bit of feedback as we go through these. We were fair to, we picked a
Nice selection of feedback, some of it negative, as well.
All right, i'm going to hand off to brie.
>> All right, so another awesome feature of cognitive serves is
Called face api. What this does is image
Processing to detect, analyze and organize faces and photos
And i'll show you right here. So what it does, it will detect
A number of faces in photos and it will return back a unique
Face id for each face it identifies, and it will return
Back facial attributes. This includes predicted gender,
Predicted age, whether you're wearing glasses or not.
As you'll see in this demo, emotion. The way that it returns emotion
Is you can see up there, as a confidence score between 0 and
1, 1 Being most confident the face detected is actually
Demonstrating that emotion, 0 the least confident.
And it will return for anger, contempt, disgust down to
Surprise, that score between 0 and 1.
So if we switch back here, you will see my face up hear, and
You'll see that it's not implemented quite yet.
I have to bend down a little bit and i'll snap a photo right now.
And you'll see analysis not implemented quite yet.
And i'll go back to the code and implement the face api.
Let's see, oh, thank you. And so it's very similar to the
Text analytics, where you create a new instance of the face api,
And we'll delete this code here. And what we want to do is we'll
Have a file stream so it is a local image file and we'll set
Up a list and we only want it to return the emotion.
Because as you saw before it can return a lot of different facial
Attributes and all we want is the emotion.
I'll go ahead and, let's see if we
Can... And so then you make the api
Call and you put in the file stream and the attributes you
Want to get returned. So then if it does detect a
Face, it will return back the first face that it finds.
And it will only bring back the emotion. As i said before, it does return
A score for all, i think it's 6 emotions so we actually just
Created a method so that it would turn back the emotion with
The highest score. So if we go ahead and implement
This now... All right, try this again.
You can see that now it is detecting my emotion, and i'll
Do a few so that you can see. I'm not an actress, so please...
[ Laughter ] let's try some
Anger. All right, maybe some neutral.
Should i do one more? how about surprise?
All right, so -- fear, all right. That's how i feel up here. So, nice.
[ Laughter ] [ applause ]
We wanted to make it a little more fun.
We did the music player earlier so we'll go ahead and set this
To music, as well. [ Music ]
All right. So we did this with local image
Files but you can also do image urls with the face api so we
Decided to make a copy of this page with twitter.
And we have already tweeted with our #mlnet demo but we'd
Like some participation from you guys. If you go ahead and take a
Selfie, take a photo of your neighbor's face, with the
#Mlnetdemo you'll get to be featured up hear.
So i'll hand it off to row wan. >> We thought we had the music
Player but we'd also do it for the twitter stream as well.
I thought we'd add some percussion.
Based on the faces coming in. [ Music with percussion ]
Of course we should bring back in everything else we had
[ ♪ Music ♪ ] join it all together in one big
Happy family.
The hashtag is #mlnetdemo.
All right, so it's just that there's ways that are easy to
Use different ml and ai solutions in your .Net
Applications. If you have any questions, we'll
Be hanging around off to the side here.
Thank you very much for joining us this afternoon.
[ Applause ] ♪ music ♪ ]