One of the most common questions that we
get asked frequently is how accurate is
your vision system. Accuracy is probably
what defines how good your vision system
is performing
Hi I'm Raghava Kashyap a CEO of qualit
as technologies and in this video I'm
going to break down all aspects of how
to measure accuracy and repeatability to
verify if your vision system is
performing to its expectations. Before we
get started let's define you know what
is the error basically the opposite of
accuracy. Error can be defined as a
deviation of your vision system results
from the ground truth in other words the
actual results. Let me say that again
error is a deviation from the vision
system result that is the result
produced by a vision system versus the
true result or the ground truth so the
key word here is really the result and
the result depends on your objectives of
the vision system let me explain that a
couple of examples so let's say the
objective of the vision system was to
measure the diameter of an engine board
so here you are actually measuring the
physical dimension of the board which is
present on the engine so in this case
the error is really the deviation or the
difference between the vision system
output that is the diameter measurement
result returned by the vision system and
the actual diameter of the system let's
take another example say the objective
of the vision system was to read a
character present on an engine chassis
and in this case the error would be the
deviation from the read output by the
vision system versus the actual
characters imprinted on the engine
chassis so in this case the value result
returned by the vision system matches
you know what is the actual printed
character making it a match a third
example say your vision system is
trained to actually look for defects and
classify a part as okay and not okay go
too bad and depending on surface defects
and anomalies present on the surface so
in this case the error is the deviation
from the vision system result that is
the judgment given by the vision system
and whether a part is okay or not okay
and the true value which is was the part
really okay or not okay
here again there are two types of errors
false negatives and false positives
false negatives are where an okay part
an actually good part is misclassified
as not okay or bad by the vision system
so here it's unnecessarily rejecting a
good part in the case of a false
positive a not okay part or a defective
part is misjudged as okay by the vision
system now as you can imagine the false
positives are the more dangerous kinds
of errors because here you're actually
miss classifying a defective part and
then passing that which has the
potential to actually reach you know go
out of a production line so in the ideal
case the false positives should ideally
be 0 ensuring that no defective parts
are classified as okay another qualifier
of your vision system is repeatability
repeatability is defined as a
consistency of judgment of the vision
system results in other words how
consistent are the results from a vision
system when you inspect the same part
over and over again repeatability is the
foundation on which error is reduced in
in other words if your system is not
repeatable and is classifying the same
part with different results there's no
way you can actually control your error
here are a few important points to keep
in mind when measuring accuracy the
first important aspect is
you have to have a solid way to measure
ground truth which is the true value so
let's go back to the example where we
talked about measurement or the diameter
now if you don't have a reliable way of
actually measuring the true value of the
diameter you're gonna go on a wild goose
chase with your vision system so we've
had a number of instances where there is
actually an error with the measurement
and which we were comparing an erroneous
ground truth and trying to match the
vision system with the ground truth so
firstly have a very rock-solid way of
defining your ground truth number two
ensure that there is enough variations
that you are testing against so if you
are measuring repeatability or accuracy
make sure there's enough variation and
you're conducting enough repeatability
yes at least five to ten for each sample
before you actually conclude that your
repeatability is you know where you
expect it to be so let's look at some
practical examples which I had brought
up earlier so say in the case of
measurement so in the example that I
shared with the diameter measurement let
let's look at how we would identify
repeatability in this case the
measurement values for three different
parts taken five times each is recorded
on this excel sheet and you can see that
the diameter variation varies between
eighty five point zero one six all the
way to 84 four point nine one five and
the true reading for this case is eighty
five point zero zero five now
repeatability would be measured by the
standard deviation of all the values
that have been taken five times for this
part and in this case it's about 63
microns
now the rough rule of thumb
is if you require an accuracy of a
certain value let's say 20 microns or
100 microns the standard deviation needs
to be about a third of your accuracy
requirement now the true reading in this
case as you can see is eighty five point
zero zero five and we have actually
tabulated the error which is the
diameter variation from the vision
system reading and the true reading and
when you consider the mean error it's
about 48 microns so in this particular
case you can see that the standard
deviation is quite high which means that
the system is really not repeatable even
though the mean error is maybe a little
bit less so you need to work on ensuring
that your repeatability or you know your
vision system calibration or your image
acquisition clarity needs to be
investigated in the second example we
when we look at the standard deviation
we see it's quite a highly consistent
or repeatable measurement system the
standard deviation in this case is only
8 microns which leads us to an error of
about 20 25 microns which is you know
quite good and in this case the true
reading was 85 point zero zero two also
leading to a mean error of 7 microns so
this is probably an ideal condition
where if you need something which is
accurate within 20 microns this
would be a good system to
use in the third example you can see
that the repeatability
repeatability is really high but the
error is also high so the deviation from
the true value is quite high now here I
would go and investigate whether the
actual true value was measured correctly
or not whether it's really eighty four
point nine you know was there an error
in the actual true value because when
you see something very highly consistent
it's usually a calibration error or the
ground truth was measured incorrectly in
the second example we were looking at
you know how the OCR values read by the
vision system compares against the
actual value so in this particular case
the same part has been inspected five
times using the vision system and you
can see that except for one part all of
them matched the actual value that was
printed so the repeatability here is 80%
now let's look at how we would measure
accuracy in this case different parts or
ten different parts where judged were
evaluated whether the vision system
matched the actual value so in other
words was the OCR string read correctly
and here you would see one out of the
ten did not read the string correctly
resulting in an accuracy of 90% the
third example is where we were judging
a part as okay or not okay depending on
defects so in this particular case a
single part was inspected five times and
the vision system
returned not okay twice and okay three
times making it a repeatability of 60%
so it's not really a consistent system
so you're seeing a lot of inconsistency
for the same part the actual value is
okay so if you look at you know the
match two of them you know did not
conform to the actual value which also
leads that there's you know some error
in the judgment so before you actually
go into measuring error you will need to
fix the repeatability before going any
further in your error correction but
assuming you did that the next
measurement that one would do is
actually look at you know how the error
was with the actual values and you can
see that if you compare the vision
output of the actual value these were
the matches so the yes meant that the
vision and output are the actual value
is matched no and if FN is a is a false
negative and an FP is a false positive
so you can see that the false positive
that means not OK values where
incorrectly judged is OK by the visual
system happened one out of the ten ten
samples that were tested and so
resulting in a ten percent false
positive rate and three out of the ten
were actually misjudged as negative or
not okay resulting in a false negative
error of 30 percent so we learned about
the definition and ways to measure
repeatability and accuracy and
practically a couple of rules to keep in
mind is for repeatability
you know the rule of thumb that we
employ is to actually have about five
different parts tested at least five
times each to ensure that the
consistency of the vision system of
repeatability is high for errors we
test for at least 100 samples and across
various permutations and combinations
depending on you know what is the
variation in terms of the actual samples
that you gonna be inspecting so that
really goes by experience the rule of
thumb is to do about a hundred
measurements each so the next time
you're in the market are starting a
vision system implementation make sure
to have a plan to objectively evaluate
your vision system both in terms of
repeatability and in terms of accuracies
ar measurement
of errors hope this video is useful
thanks for watching and have a great day