you're not going to want to miss this episode of the ai show where we talk about how to automate
custom vision model creation with automl for images make sure you tune in
hello and welcome to this episode of the ai show we're going to talk about how to automate custom
bidding vision model creation with auto ml for images with sarah stairs how you doing my friend
hello seth how are you fantastic so tell us who you are and what you do bud
hello so i am based in uh in paris in france i am serge rodkovsky working for microsoft since
three years now and i am mainly working on ai projects including computer vision
that's fantastic but you you can actually work on real world problems not like the stuff that
i kind of make up right yes but that's correct so currently many companies and organizations
are using computer vision for multiple kind of projects like predictive maintenance healthcare
security retail autonomous driving and so on so the good thing is for computer vision is that it's
very easy with automl for images available with azure machine learning to build and train some
new models to deliver to deploy this model into azure and then to quickly uh do some predictions
based on images because computer vision is not only is not analyzing content from database
or database management system you will analyze images and images is just a collection of pixels
and it's very easy to build and trend some deep neural network in order to help companies and
organizations to take advantages of images or videos i mean that's fantastic so you're
going to show it you're going to talk about three world cent world uh real world scenarios can you
describe them for us yes for sure so let's take some examples so the first examples i would like
to share with you is how to use computer vision model for quality control so quality control
is key many companies and organizations are doing predictive maintenance package inspection
product and component and supply defect reduction improving worker safety and many additional things
so in this case the first example will be about quality control how to build and train a model an
object detection model with azure machine learning and using automl for images so i will quickly
show you some examples i have another example in the retail industry in order to count product from
shelves and the last example will be about how to detect fire from images and fire from a webcam
from example awesome well can we take a look yes so let's start with the first one so as
you can see this is a python notebook i have previously built a custom vision model to detect
quality control so as you can see sometimes there are quality control issues in this example
this is not correct and on the right part this is correct the beauty of photoml for images is that
it will automatically choose for you the best custom vision model and the the best
repair parameter tuning parameters directly using automl for images so first i tried to
to analyze a collection of images i was able to run automl for images to build and trend my model
and then to deploy my model and when new model is deployed it's very easy to use it so let's take a
quick example okay as you can see this is just an image of a product and as you can
see there is a gap here between these two blue pieces so this is not correct according to this
image let's have a look if our model is able to detect this so i'm going to run live all the
all the steps and as you can see yes this is the first result there is a an issue with
a package inspection it says not correct with a confidence close to 80 percent which is very good
and by the way this is the location of the region where the computer vision model is analyzing
the content so what do you think that's cool i mean like looking at it um did you have to do
much to get this to work or like i see that the inference is happening but how did you how did
you build the model like we talked about ottawa from images a couple of weeks ago is it just a
question of giving it the images and letting it go yes so there was a dedicated ai show
about automl for images and the good thing is that with automl for images you can do object detection
program you can do image classification problem instant segmentation as well so what i previously
did was to collect my images to tag my images and then with just a couple of lines like that one i
can do an object detection using automl for images using in this case i am using yellow version five
which is very good for objection can you do me a favor press control plus a couple times just
to make the code a little bit bigger uh control plus there you go and then if you scroll down
i see okay i see so basically and we i know we did a show on this but seeing a real world case
scenario is actually kind of cool you basically tell it object detection you're using yellow v5
and you say go yes i said just say go of course it's always possible to add additional information
options parameters but just with a couple of lines just say okay i want image in object detection
and i can pick up a framework like euro v5 and that's it yeah that's that's really and how long
did it take to train the the right model is it like oh in this model it was a question of 10
or 15 minutes okay based on 100 images not so much okay but it's a good starting point
and the quality of the model was quite very good uh based on my validation data set and
and that's cool because going from from an idea to a prototype of machine learning in 15 minutes
i think is quite cool if you ask me all right what's the next example yes it is quite cool so as
you can see this is a result and what you can do it's always good to combine for example a speech
to text if you want the quality control prediction is not correct with a competence of 79 percent
so this is just the use of of our speech to text capabilities so you can play this
audio file directly from the interface or you can send a message to a team's channel
so of course you can integrate teams with azure machine learning and you can send a message to a
team channel let's have a look you will see there that the quality control prediction is not correct
so now you're combining a bunch of cool ai things to to alert people if there's something wrong
because there might be there might be a situation where it's imperative that someone get a message
about whatever it is that's going on with the picture and there's a question all right that's
correct so the goal is in real time to send some alerts so you can send emails if you want to or
just use a team's notification because with just a couple of lines you can do it
only when there is something wrong in this case when this is not correct
that's cool all right what are the other examples i know i jumped the gun but this is really cool so
far next example it's about retail okay so we know that all the retailers have some issues with uh
with their products on the shelves and they want to be able to uh to detect if products are missing
for example for the resupply and this is another example so it's still about an object detection
custom vision model for that but in this case as you can see i will be able to
count the number of drinks and i can't say okay there is uh no more water or there is too much
sodas or whatever and it's very easy to do let's go to the key part so this is
another new image okay that you can take from a camera or a webcam for example and let's run the
model and the model they will say okay there is 18 drinks coca and fizzy water and 7 up and whatever
okay and same thing you can automatically uh count the number of products and in this case
it will say i need to resupply six drinks because some drinks are left and same thing i can send a
message on my team's channel and i know that i i can resupply some products oh my goodness this is
i mean this is really cool because you can imagine a scenario where you literally have a camera
pointed at the stock and then every you know i don't know every night it just checks and it's
like hey you know you need to put another another seven up in there or something that's it that's
pretty cool okay there is uh not so much things to do except building a custom vision model to detect
the brands and then it's very easy with automl for images and all the great options there to create
to build and train and to deploy your model and one thing you can deploy your model on the edge
not only on the cloud but you can deploy it on the edge directly into a webcam for example
or cctv interesting so the model that you build in auto ml for images you can download and put it on
any device that could run a model like that okay that's cool all right what else you got for us bud
okay so next example is how to create and deploy your model on the edge let's say you have a cctv
and you want to be able to detect any new fire for fire detection so same thing i am using an image
classification i have built and trainer model using thousands of images of fire and non-fire
i was using as well auto ml for images from azure machine learning and then i deployed
the model and what we can do first is just to play a little bit with a model just to
see if we can detect a fire so this is a an image from a fire it was in paris a couple of weeks ago
the good thing is that nobody was hurt so that's that's great thank you and uh this is
uh we can see a fire obviously and let's call the model just to check and the mother says yes there
is a fire with 81 percent okay wow so imagine that you build an trainer model on the cctv on a webcam
and in real time it will be able to detect a new fire okay yeah and what you can do as well using
integration with uh additional services is that let's say you have a fire department address okay
so you can retrieve the latitude and longitude so it's an address in paris as you can see and
this is the webcam location the fire was in place de opera in paris so you can retrieve as well
latitude and longitude and you can use bing to to collect information like uh the uh the the time
for the fire department to go to the fireplace and in this case it will say okay
usually it takes 14 minutes but at the moment we have a lot of traffic jams and
the additional travel time due to traffic will be 99 minutes and the good thing is that you can
you can send a message to the fire department this is the map so you can take this through it and
then go to boulevard valid and so on and you can use a big integration in order to display in a map
the location between the fireplace okay so we have a lot of traffic in place developer at the moment
and this is the the shortest destination in order for the fire department to go to the fireplace
so another example what you can do of course is to analyze in real time videos okay let's have a
look in this case so let's play okay so as you can see at the beginning of the video there is no fire
detected okay and then after a couple of seconds you can see a car who is currently burning okay
so let's try to analyze this video using our custom vision model and i can i can
run the process it will analyze some frames at the beginning no fire was detected
and now after this frame there is an alert and same thing i can have a notification
on my team's channel as you can see wow fire detected at frame 240 270 and so on so i have
uh real-time alerts and as you can see yes at the beginning from my cctv there was no
fire with 78 percent of confidence and after a certain period of time 100 percent of fire
yes because obviously the model was not wrong at this time and that's it so another nice example
about how to use a computer vision model in order to detect some specific events it can be a car
accident it can be smoke detection it can be fire detection and you can run your model directly on
the edge directly on the video camera so what do you think i the thing i like about this the most
is that i mean i i make ai models all the time but i look at it from a clinical perspective like is
this model working is is everything doing the right thing i like the way especially with the
fire one and obviously if fires are really sad and if anyone had to go through it super sad
but imagine a scenario where fire departments can be alerted immediately that there's a fire
somewhere and it tells them how to get there the fastest in the same instant i think i think
putting that all together is actually really cool how did you come up with these scenarios search
it was just a a curiosity what is good with ai is that of course you can spend
some time to build a model but think about a full integration with any solution like a cctv or like
any specific equipment what you want is not only to predict something like the fire or or to do uh
any kind of question maybe you want a little bit more you want real time notification
okay and if you need to go from a place to another one if you can have the shortest uh
how to say uh path or what that's nice and the shortest will depend on the time of the day
it will depend on the traffic jam and using integration with bing
you can have everything included the forecast for example so at the moment there is no rain most
recently and you can have additional information so you can create a end-to-end solution
with a combination of multiple azure services and of course including uh our custom vision model
which is key in this kind of application to detect the fire or any specific event i love it so where
can people go to find out more my friend sorry where can people go to find out more about how
to do this so if you want to learn a little bit more of course there is the official documentation
from automl for images and i made a dedicated github repo where you can find more than 40
python notebooks all of them are available for for free i mean so if you want to evaluate to ml
for images you are not you can use or leverage my uh demos notebook if you want to uh to to
start uh evaluating or to ml for images awesome well sarge thank you so much everyone go check
out those notebooks they're really cool thank you so much for spending some time with us my friend
you've been learning all about how to automate custom vision model creation with automl for
images with search thank you so much for watching and hopefully we'll see you next time take care