if you clicked on this video you are certainly interested in computer vision applications
like image classification image segmentation object detection and more complex tasks like
face recognition image generation or even star transfer application as you may already know
with the growing power of our computers most of these applications are now being realized using
similar deep neural networks what we often refer to as artificial intelligence models
there are of course some differences between the deep nets used in these different vision
applications but as of now they all use the same basis of convolutions introduced in 1989 by yan
loken the major difference here is our computation power coming from the recent advancements of gpus
to quickly go over the architecture as the name says convolution is a process where an original
image or video frame which is our input in a computer vision applications is convolved using
filters that detect important small features of an image such as edges the network will autonomously
learn filter values that detect important features to match the output we want to have
such as the object's name in a specific image sent as input for a classification task these filters
are usually of size 3x3 or 5x5 pixel squares allowing them to detect the direction of the edges
left right up or down just like you can see in this image the process of convolution makes a dot
product between the filter and the pixels it faces it's basically just a sum of all the filter pixels
multiplied with the values of the images pixels at the corresponding positions then it goes to
the right and does it again convolving the whole image once it's done these convolved features
give us the output of the first convolution layer which we call this output a feature map
we repeat this process with many other filters giving us multiple feature maps
one for each filter used in the convolution process having more than one feature map gives
us more information about the image and especially more information that we can learn during training
since these filters are what we aim to learn for our task these feature maps are all sent into the
next layer as input to produce many other smaller sized feature maps again the deeper we get into
the network the smaller these feature maps gets because of the nature of convolutions and the more
general the information of these feature maps become until it reaches the end of the network
with extremely general information about what the image contains disposed of our many feature maps
which is used for classification or to build a latent code to represent information present in
the image in the case of a gan architecture to generate a new image based on this code which we
refer to as encoded information in the example of image classification simply put we can see
that at the end of the network these small feature maps contain the information about the presence of
each possible class telling you whether it's a dog a cat a person etc of course this is super
simplified and there are other steps but i feel like this is an accurate summary of what's going
on inside a deep convolutional neural network if you've been following my channel and posts
you know that deep neural networks proved to be extremely powerful again and again but they also
have weaknesses and weaknesses that we should not try to hide as with all things in life deep
nets have strength and weaknesses while strengths are widely shared the latter is often omitted or
even discarded by companies and ultimately by some researchers this paper by alan yule and chenxileo
aims to openly share everything about deep nets for vision applications their success and the
limitations we have to address moreover just like for our brain we still do not fully understand
their inner workings which makes the use of deep nets even more limited since we cannot maximize
their strength and limit weaknesses as stated by o hobart it's like a road map that tells you where
cars can drive but doesn't tell you when or where cars are actually driving this is another point
they discuss in their paper namely what is the future of computer vision algorithms as you may
be thinking one way to improve computer vision applications is to understand our own visual
system better starting with our brain which is why neuroscience is such an important field for ai
indeed current deep nets are surprisingly different than our own vision system
firstly humans can learn from very small numbers of examples by exploiting our memory
and the knowledge we already acquired we can also exploit our understanding of the world
and its physical properties to make deductions something that a deep net cannot do in 1999 gupp
nick ital explained that babies are more like tiny scientists who understand the world by performing
experiments and seeking causal explanations for phenomena rather than simply receiving stimulus
from images like current deep nets do also we humans are much more robust as we can easily
identify an object from any viewpoint texture it has occlusions it may encounter and novel context
as a concrete example you can just visualize the annoying captcha you always have to fill in
when logging into a website this captcha is used to detect butts since they are awful
when there are occlusions like this as you can see here the deep net got fooled by all the examples
because of the jungle context and the fact that a monkey is not typically holding a guitar this
happens because it's certainly not in the training data set of course this exact situation might not
happen very often in real life but i will show some more concrete examples that are
more relatable and that already happened later on in the video deep nets also have strength that we
must highlight they can outperform us for face recognition tasks since humans are not used to
until recently seeing more than a few thousands of people in their whole lifetime but this strength
of deep nets also comes with a limitation where these faces need to be straight centered clear
without any occlusions etc indeed the algorithm could not recognize your best friend at the alwyn
party disguised in harry potter having only glasses and a lightning bolt on the forehead
where you would instantly recognize him and see whoa that's not very original it looks like you
just put glasses on similarly such algorithms are extremely precise radiologists if all the settings
are similar to what they have been seeing during their training they will outperform any human
this is mainly because even the most expert radiologists have only seen a fairly small
number of ct scans in their lives as they suggest the superiority of algorithms may also be because
they are doing a low priority task for humans for example a computer vision app on your phone
can identify the hundreds of plants in your garden much better than most of us watching the video can
but a plant expert will surely outperform it and all of us together as well but again this strength
comes with a huge problem related to the data the algorithm needs in order to be this powerful
as they mentioned and as we often see on twitter and article titles there are biases due to the
data set these deep nets are trained on since an algorithm is only as good as the data set it
is evaluated on and the performance measures used this dataset limitation comes with the
price that these deep neural networks are much less general purpose flexible and adaptative
than our own visual system they are less general purpose and flexible in the way that contrary to
our visual system where we automatically perform edge detection binocular stereo
semantic segmentation object classification scene classification and 3d depth estimation
deep nets can only be trained to achieve one of these tasks indeed simply by looking around
your vision system automatically achieves all these tasks with extreme precision where deep
nets have difficulty achieving similar precision on one of them but even if this seems effortless
to us half of our neurons are at work processing the information and analyzing what's going on we
are still far from mimicking our vision system even with the current depth of our networks
but is that really the goal of our algorithms will it be better to just use them as a tool to improve
our weaknesses i couldn't say but i am sure that we want to address the deep nets limitations
that can cause serious consequences rather than omitting them i will show some concrete examples
of such consequences just after introducing these limitations but if you are too intrigued
you can skip right to it following the timestamps under the video and come back to the explanation
afterwards indeed the lack of precision we previously mentioned by deepnets arises mainly
because of the disparity between the data we use to train our algorithm and what it sees in real
life as you know an algorithm needs to see a lot of data to iteratively improve at the task it is
trained for this data is often referred to as a training data set this data disparity between
the training data set and the real world is a problem because the real world is too complicated
to accurately be represented in a single data set which is why deep nets are less additive than our
vision system in the paper they call this the combinatorial complexity explosion of natural
images the combinatorial complexity comes from the multitude of possible variations within a natural
image like the camera pose lighting texture material background the position of the objects
etc biases can appear at any of these levels of complexity the data set is missing you can see
how these large data sets now seem very small due to all these factors considering that having only
let's say 13 of these different parameters and we allow only 1000 different values for each of them
we quickly jump to this number of different images to represent only a single object the current data
sets only cover a handful of these multitudes of possible variations for each object thus missing
most real-world situations that it will encounter in production it's also worth mentioning that
since the variety of images is very limited the network may find shortcuts to detecting
some objects as we saw previously with the monkey where it was detecting a human instead of a monkey
because of the guitar in front of it similarly you can see that it's detecting a bird here instead of
a guitar probably because the model has never seen a guitar with a jungle background this is called
overfitting to the background context where the algorithm does not focus on the right thing
and instead finds a pattern in the images themselves rather than on the object of interest
also these data sets are all built from images taken by photographs meaning that they only cover
specific angles and poses that do not transfer to all orientation possibilities in the real world
currently we use benchmarks with the most complex data sets possible to compare the current
algorithms and rate them which if you recall are very incomplete compared to the real world
nonetheless we are often happy with 99 accuracy for a task on such benchmarks
firstly the problem is that this one-person error is determined on a benchmark data set meaning that
it's similar to our training data set in the way that it doesn't represent the richness of natural
images it's normal because it's impossible to represent the real world in just a bunch of images
it's just way too complicated and there are too many situations possible these benchmarks we use
to test our data set to determine whether or not they are ready to be deployed in the
real world application are not really accurate to determine how well it will actually perform
which leads to the second problem that is how it will actually perform in the real world let's
see that the benchmark data set is huge and most cases are covered and we really have 99 accuracy
what are the consequences of the one percent of cases where the algorithm fails in the real world
this number will be represented in misdiagnosis accidents financial mistakes or even worse
death such cases could be a self-driving car during a heavy rainy day heavily affecting
the death sensors used by the vehicle causing it to fail many depth estimations
would you trust your life to this partially blind robot taxi i don't think i would similarly would
you trust a self-driving car at night to avoid driving over pedestrians or cyclists where even
yourself had difficulty seeing them these kinds of life-threatening situations are so broad
that it's almost impossible that they are all represented in the training data set and
of course here i use extreme examples of the most relatable application but you can just imagine how
harmful this could be when the perfectly trained and tested algorithm misclassifies your ct scan
leading to misdiagnosis just because your hospital has different settings in their scanner or because
you didn't drink enough water or die anything that would be different from your training data could
lead to a major problem in real life even if the benchmark used to test it says it's perfect also
as it already happened this can lead to people in underrepresented demographics being unfairly
treated by these algorithms and even worse this is why i argue that we must focus on the task where
the algorithms help us and not where they replace us as long as they are that dependent on data
this brings us to the two questions they highlight how can we efficiently test these algorithms to
ensure that they work on these enormous data sets if we can only test them on a finite subset
and two how can we train algorithms infinite size data sets so that they can perform well
on the truly enormous datasets required to capture the combinatorial complexity of the real world
in the paper they suggest to rethink our methods for benchmarking performance and evaluating vision
algorithms and i agree entirely especially now where most applications are made for real life
users instead of only academic competitions it's crucial to get out of these academia evaluation
metrics and create more appropriate evaluation tools we also have to accept that data bias exists
and that it can cause real world problems of course we need to learn to reduce these biases
but also to accept them biases are inevitable due to the combinatorial complexity of the real world
that cannot be realistically represented in a single data set of images yet thus focusing
our attention without any play of words with transformers on better algorithms that can learn
to be fair even when trained on such incomplete data sets rather than having bigger and bigger
models trying to represent the most data possible even if it may look like it this paper was not
a criticism of current approaches instead it's an opinion piece motivated by discussions with
other researchers in several disciplines as they state we stress that views expressed in the paper
are our own and do not necessarily reflect those of the computer vision community but i must say
this was a very interesting read and my views are quite similar they also discuss many important
innovations that happen over the last 40 years in computer vision that is definitely worth
reading as always the link to the paper is in the description below to end on a more positive note
we are nearly a decade into the revolution of deep neural networks that started in 2012 with alexnet
and the imagenet competition since then there has been immense progress on our computation power
and the deep net architectures like the use of batch normalization residual connections
and more recently self-attention researchers will undoubtedly improve the architecture of deep nets
but we shall not forget that there are other ways to achieve intelligent models than going
deeper and using more data of course these ways are yet to be discovered if this story of deep
neural networks sounds interesting to you i made a video of one of the most interesting architecture
along with a short historical review of deep nets i'm sure you'll love it thank you for watching