hello and welcome back to another MATLAB video my name is patan Barr and today's
topic is semantic segmentation we're gonna cover two things number one what
is semantic segmentation number two how do you do it now before we get started I
want to mention that this video is going to cover higher-level concepts I'm not
gonna be going through a specific code example but if that's what you're
looking for go ahead and click the link in the description below that's probably
where you want to be but if you're new to this you want to learn about the
workflow or just rehash some of the concepts this videos for you so let's
start with the basics what is semantic segmentation semantic segmentation is
the process of taking an image and labeling each pixel in that image with a
certain class let me show you what I mean here is a picture of me at the
beach now don't be distracted by how fantastic I look but after going through
the semantic segmentation workflow that picture would look like this notice how
there's two classes there's person and there's background the pixels inside of
my Silhouette are assigned to person and the pixels
outside my Silhouette are assigned to background now the key to semantic
segmentation is that the computer actually did the segmenting of that
image automatically in order to do that we have to take a bunch of images of
people standing in backgrounds and we have to label where the people are and
label where the background is and then we use that information to create a
convolutional neural network and we train that network and when we put in a
new image of a person standing in some background it'll be able to segment that
image into the two parts it's pretty cool but now you're probably wondering
what's the practical application of this let me give you an example let's say
that I owned a car company called pitambar's cars and this car company
sold cars with automated driving systems let's talk about that automated driving
system one way I could create it is with semantic segmentation I could take a
bunch of images of the road and label where the lanes are
where other cars are where pedestrians are where trees are where the sky is
where the lamppost says anything I could just label all of these things and I
could do that for thousands of images and then I could use all of these images
to create a convolutional neural network that when given a new image of the road
it can pick out where the lanes are where are people where are trucks where
are buses etc etc and from there I can implement safety features like automatic
braking or that thing that stops you from turning when there's a car right
next to you or a parallel parking system obviously that requires like more
algorithms but semantic segmentation is the crux for determining where those
things are okay I hope at this point that you're convinced on why we're
focusing on the subject now what I want to do is walk you through the steps at a
very high level so let's jump into MATLAB and talk about our first step
which is labeling the images I'm going to open up MATLAB and I'll start by
opening the image label or app there's a section here called apps and we're gonna
go down and click image labeler we'll see this new screen pop up the first
thing we'll do is import our images so I'll go to load and I'll add in these
images this first picture is me as a freshman in college and the second one
is me as a senior clearly my hair grew faster than my sense of style now in
order to do semantic segmentation we first need to define our labels and
there's a section here that says define new ROI label our oh I stands for region
of interest by the way I'm gonna click that and we're gonna label eyes nose
ears and mouth I'll create a label that says eyes and I have two options
rectangle and pixel label I want pixel label because that will allow me to be
really specific with my selection and that's what I need to pick to do
semantic segmentation I'll add a description and click OK I'll create
another ROI label called ears and another one called nose and finally
mouth and once I've created all of my ROI labels I'm going
start labeling so let's zoom into my eye and start drawing a mask around it we'll
finish our selection by clicking where we started we can see that the eye now
gets highlighted let's pan over and do the same thing for my second eye now
I'll intentionally mess this one up so I can show you some of the other tools
oops it looks like I've overshot this let me use the eraser to erase the part
I don't want Oh No but now I've erased too much what will I do I know I'll use
the brush and I'll fill in the part I actually want once I finish with the
eyes I'll click the nose label and ylabel the nose then I'll do the mouth
and the ears and the same thing for the second image if I did this correctly
I've turned both pictures of myself into pictures of clowns
just kidding these are the labels while I'm doing this process for only two
images you should really be doing this for way more images depending on your
images you might be able to implement some kind of algorithm so right here
there's a button that says select algorithm click it and you can click on
create new algorithm so once you've finished labeling all of your images you
need to export your labels so I'll go here and I'll click to workspace and
when I do this and go back to MATLAB I now have a ground truth object called G
truth this object has all of the information that I gave my computer when
I labeled the images alright let's cut back to the video and
I'll teach you how to do the remainder of the steps at a high level alright
step one was to label our images and we just did that step two is to create two
data stores in image label data store and a pixel label data store now this is
very simple to do but I'm not gonna go into code so again I'm gonna direct you
to the example which you should totally check out after watching this video step
three is to partition your data stores into training sets and test sets the
reason we do this is because whenever we work with convolutional neural networks
we need a set of data to train our network on and a set of different
data to test our network on and see if it works
and that brings us to step four which is to import a pre train model into MATLAB
and modify it to be a segment again I'm gonna refer you to the code example for
more details we use a model called vgg 16 and we very easily convert that into
a segment finally step 5 train and evaluate your network and that's it
so with those five steps you've now learned the crux of semantic
segmentation I'll go over them again and I'll list them right here as well step 1
label your data or search the internet for label data step 2 create two data
stores image label data store and a pixel label data store step 3
partition your data stores into training sets and test sets step 4 import a pre
train model and modify it to be a segment and step 5 train and evaluate
your network and there you have it folks semantic segmentation now thank you guys
for tuning in really appreciate it if you have any feedback leave it below
otherwise check out the code example and I'll see you guys in another video