So, let's say we have something like 26 applications
in a benchmark suite. How do we summarize the
overall performance of a machine on the entire suite?
So we want a single number that represents the performance
of the machine. Usually, we want to use something
like the average execution time. So let's say we have
a computer X and a computer Y, a benchmark
application A, B and C and let's say that computer
X takes 9 seconds here, 10 seconds here
and 5 seconds here, whereas the computer Y
takes 18 seconds here, 7 seconds here, and
11 seconds here. The speed up of machine
X over machine Y on each of these applications is two for application A
because X is doing it in half the time as Y, 0.7 on application B because X is
actually slower than Y there. And here is 2.2 because X is more than
twice as fast as Y. Now let's compute the average execution time. Here we have
9 plus 10 plus 5 over 3, gives us 8 seconds. So on average, computer X finishes
one of these three applications in 8 seconds. On average, computer Y finishes
one of these three in 18 plus 7 plus 11 over 3, 12 seconds.
If we now compute the speedup of X over Y, on average it will be 1.5 when
computed this way. But if we average the speedups here, we get 1.63. So if we
want to use average execution times, we can
not simply average the speed ups on individual
applications to get the average speed up. Pretty
much the speed up using average execution times
is not the same as the average of
speed ups on these individual applications. If we want
to be able to average speed ups, then we need to use geometric means for both of
our application execution times and for our speed
ups. In that case we get a geometric mean
of 7.66 here, 11.15 here, and 1.456 here. So we have used the geometric mean for
the execution times for each of the computers. If we use the geometric mean for
these speed ups, we will get the same value here. The geometric mean is computed
by multiplying out, for example, 9 times 10
times 5, and then taking the nth root,
in this case 3rd root of the value, to
get the geometric mean. And because things are being
multiplied out and then the nth root taken you
can see how, if we divide these things, we will
get that the speed of computed geometric means of
execution times will be the same as the mean of
the speed ups, because we have third root of
18 times 7 times 11. These two divided is the
same as third root of 9 over 18, times 10 over 7 times 5 over
11. So as you can see this is the geometric mean of execution times, then we
can compute the speedup using them, and that's
the same as the geometric mean of individual
speed ups. However this only works for geometric
means. We do prefer average execution time when
we're computing the average times. However, just
you must be careful not to compute individual
speed ups and then average them. Instead,
you should be computing the average execution times
and then computing the speed up using
those averages. You should also remember that simple
average should not be used on ratios of times. So basically a speed up here is
a ratio of two times. We should not be using a
simple arithmetic mean on these values
because they're ratios. Geometric means, on
the other hand, can be used on ratios, but, for this to
work, we also have to do the geometric mean on times, too.