>> You're not going to miss this episode of AI Show.
We talk about applied AI,
Computer Vision, and OCR.
Makes sure you tune.
[MUSIC]
>> Hello and welcome to this episode of the AI Show.
We're talking about applied AI, Computer Vision,
and Optical Character Recognition.
Let's start with introductions.
My friends welcome.
Tells us who you are and what you do.
We'll start with you Carlotta.
>> Hi, everyone. I'm Carlotta.
I'm based in Italy and I'm a Cloud Developer Advocate at Microsoft
focus it on AI machine learning
technologies and I'm pleased to be here today.
>> Fantastic, Bethany.
>> Hi everyone. I'm Bethany Jepchumba from Nairobi, Kenya.
I work with Carlotta as a Cloud advocate for AI
and machine learning and I'm also thrilled to be here.
>> Fantastic. Why don't you explain what all of this is all about.
Bethany, we'll go to you, my friend.
>> Today we're talking about computer vision and
this is a subset of AI that deals
with giving applications the ability
to see the world and be able to make sense of it,
turning it into either readable or audible experiences.
As you can see in the image on the right,
AI sees an image as a group of pixels values,
and it can be able to interpret it,
giving you an idea of what exactly is in the image.
It can do things like analyze your image, recognize text.
That is what we're talking about in OCR,
Optical Character Recognition, and
finally detects the different objects in an image.
>> It's basically anything having to do with
pictures and getting computers to do something with it.
Am I right?
>> Yeah, you can also add videos to the mix.
>> Oh, see. I knew this,
but I feel like there's a ton of
value here with what people can do,
but can you give us like in practice,
what are some things that one can do today?
>> For example, using the Azure Computer Vision service,
you don't even need to write any line of code,
but you can use the API to do things
like detect images in objects.
There's this app called the seeing AI,
and the application actually reads out loud what is in
the environment and people who are not able to see,
can be able to get on
audible experiences of what is going on in the environment.
>> I love it. Is there a way that we can take
a look at what this looks like in action?
Carlotta, can you help us out with that?
>> Sure, of course.
I would like to show you a short demo in a way to guide you
on how you can integrate
your solution Computer Vision Service and in particular,
I'm going to show you how you can integrate
the Computer Vision service in a React Static Web App,
which we'll call the service using a URL
of an image type from the user as input.
Also I will show you how to deploy it on the Cloud.
As prerequisites of this demo,
you will need two types of Azure resources,
a Cognitive Service Resource type,
which we will need to consume
the service and a Static Web App resource,
which is a service that automatically builds and deploys
full stack web apps to Azure from GitHub code repository.
Each time you push comets or acceptable requests,
in a branch of your choice.
>> This is awesome.
>> Yeah, sorry. Let's go maybe into the code itself.
I have it in my visual studio code environment
and the functionality which calls the Computer Vision APIs,
is hosted in a separate file.
You can see here a JS file in which at
the bottom we start with configuring the key and the endpoint,
which are the parameters used to
consume the Computer Vision Service.
Then the visual features we are going to
use to call the service as well.
Scrolling down, you can see the main function of this JS file,
which is the computer Vision function.
This function performs a few tasks.
For example, it initialize
the client using the key and then point previously set.
Then it initialize the URL of the image we are going
to analyze and it is going to call
the analyze image API using the visual features and
URL we defined before the computer vision client,
of course, and the analyze image of
the Computer Vision SDK for JavaScript.
In addition to these,
if our image that we want to analyze contains text,
check that we do by checking if in
the return list of tags we have the handwriting or text tags.
In this case, we are going also
to call the read texts from URL function,
which is going to call the Read API
of the Computer Vision Service function,
which has optical character recognition capabilities.
It is able to extract text from the image.
But I also would like to show you,
when you have these function defined in a separate JS file,
how you can integrate in your React Static Web App, the service.
In a few lines of code you can call the Computer Vision
function you just see
in the other file and reset the state of the app.
We also are going to
display the results of the Computer Vision Service on
our application by converting
the results from a JSON format to a readable format.
Once you have all these things set up,
the next step is of course running the app.
The best practice is running the app
locally and then if everything works fine,
deploying it on the Cloud,
but for time constraints today,
I'm going to show you directly how you
can deploy these app to the Cloud.
We are going to trust this code.
In order for our app to retrieve
the key and endpoint of the service,
we're going to add a pair of lines of
code which tells to our app that
the key and endpoints parameters can be
retrieved from the secrets of our GitHub repository,
in particular, in the action section.
I've already set up these secrets for you.
In the Yamo file of the GitHub workflow file,
I'm going to add this pay of line of codes here to tell our app
that can retrieve the key
and endpoint starting from the secrets of the GitHub repository.
Then we can just commit and
push these changes because as I told you before,
by pushing in these keys to main branch these edits,
we are going to launch a workflow.
We're going to run a workflow
which will finally deploy the app on the Cloud.
I'm going to, for example,
right here add secrets to actions as a comic message and I'm
going to commit and push these changes here.
I will show you how these results.
Here we are. This is my GitHub repository.
These results in a workflow running
here and deploying the app on Azure.
>> Now I will hand over again to Bethany,
I guess in order to show us these app running on
the Cloud and the Computer Vision response to our image.
>> Just a couple of questions before you go. This is awesome.
Basically, if I'm understanding right,
this is all a client-side reacts Static Web App.
It's just a demonstration on how to
get it started really quickly with the API
that then you put the keys into GitHub Action and then pushes
it to Azure Static Web Apps. Am I getting this right?
>> Yeah, it's perfect. Definitely, yeah.
>> Fantastic. Then the other question I have is
generally I start good practice to
put the keys in the client side,
but this is just an example for those that are watching,
you generally want to put that part in the backend and the
function. Am I understanding that right?
>> Yeah, of course, it's right.
This is a very simple example just to
familiarized with Computer Vision Integration your solution,
but definitely, this is the right way to do things.
>> I love it and that's a cool thing about JavaScript.
If it's in the backend and frontend,
it looks the same.
Really good example. Bethany,
you're going to show us these in action. Is that right?
>> Yeah, that's something I was going to do in a few.
I will use this image and I'll copy the image link,
go to our tab and enter the URL.
Once I enter the URL here,
it will go ahead and analyze.
While it's analyzing,
therefore, key things I'm going to show you.
The first one is the color aspect of it.
As you can see, it's given me
the URL and the different categories they belongs to.
But I want to focus first on the color.
In the color gives me three major things.
Number 1, it gives me an idea of
whether the color is either black and white image or not.
In our case, it is not.
The other thing is the dominant foreground color
and the dominant background color,
as well as the accent color.
As you can see, it gives you an idea of if you
want to build an app with this color-code,
it gives you an idea of what color scheme to follow.
The second one is tags.
Tags give you an idea of what are the contents of the image.
The tags here are texts.
It's a font, it's a circle,
there's a couple of logos,
there's clip arts, graphic design.
This is giving you
random things that can be used to describe the image.
The third one is the caption and the description of the image.
The description here is this is a diagram
and the confidence level is not that high.
It's telling us, yes,
this is a diagram,
but of course,
there's so many things around it.
It's not just a diagram.
The last part is OCR,
being able to recognize the text here.
The text, as you can see is make things happen.
How are you able to recognize the text?
Let's see from our results.
The first one here, as you can see,
the model value of this is a text and it's reading the results.
It's giving us a bounding box or where they are,
and the first one is a text make,
and this is the location of the texts,
and the confidence level is 96 percent,
that is very high.
Then coming down here,
it's the text things giving us the location of it,
and also the confidence level is about maybe three percent.
The last word is happen.
This is the location of the word and
even has a higher confidence level.
It's about being able to
detect the image and figure out what are the texts there,
what tags can you associate with the image, and so on.
>> This is cool because this isn't the first time I've seen these,
but every time I see the results of calling a cognitive service,
it's amazing from design.
You showed us the colors to the text that's in there,
to any tags that might be associated.
There's so much information that you can get,
just by calling a service.
I think it's really cool, Bethany.
>> Yes, so many. This is just like
scraping the surface on what you can do.
There's so many other things in case you want
to detective if this is a famous person.
There's also a tag for that.
>> That's really cool. I just decide story since we're all here.
I was on stage with a guy that was famous and I was showing
this service off and I made fun of the fact
that it recognized him but not me,
and so I guess if Devs are watching,
and so now I'm a famous person for cognitive services.
>> You talked to the cognitive service team?
>> Isn't that cool? No. If you put a picture of me,
it recognizes me because I made fun of the Devs and they
got me back by making the AI famous. I'm so happy.
>> Can you link me up to that Dev?
>> Yeah, I should send you,
it's really funny because I tried to make that joke every year
at a conference and it stopped working,
and they're like we got you, Seth.
It's actually really cool the amount
of people that it recognizes,
the amount of information you get from design.
But serious, there's so much to learn in this space.
Like if, let's just say someone's watching and they're beginner,
what would you suggest they should do OCR audio?
>> Yeah, we have wonderful curricula actually for AI beginners.
For example, if you want to learn
more about the behind the scenes of these Computer Vision Service.
Which algorithms are behind the scenes of these service?
How are these algorithms?
We are talking about neural networks, deep learning,
how these algorithms work,
and how they can see the words.
If you are interested in these topics,
definitely look at these AI for beginners curricula.
>> Fantastic. I'm going to put a link below.
It's just you go to ak.ms AI for beginners repo.
I think it has lessons,
it has explanation,
it has really cool diagrams. Am I getting this right?
>> Yeah, right. It's divided in different lessons,
so it's very useful also if you want to use it on your own,
let's say to learn it on your own.
>> Fantastic. For those that want more of
a lessons guided something.
Is there something like that for them?
>> Yeah, of course.
As Bethany just said,
we just scraped the surface
of the capabilities of Computer Vision Services.
But there are more.
If you want to learn more,
there is a learning path for Microsoft
learn dedicated to Computer Vision Service.
Definitely, you should catch it up.
>> The last thing we'll go to you, Bethany,
is this code that you showed,
is it available somewhere?
>> Yeah. There's a link that we shared,
and you can go to that link and get access
to the code and play around with it,
see if you can make your AI better also.
>> I think when it comes to you,
I am the worst and so I will not be able to help you,
but maybe some enterprising that do AI will be able to do that.
Thank you so much for spending some time with me my friends.
>> Thank you very much.
>> We've been learning all of applied AI, Computer Vision,
and Optical Character Recognition here on the AI show.
Thanks for watching and hopefully,
we'll see you next time. Take care.
[MUSIC]