- Yeah, so this is kind of funny paper
and presentation from me because my background
is not in image compression, not in the least.
But bear with me, and see if any of this
makes sense to you.
So, basically the setting for this was
we've been working on our RoomAlive Toolkit,
something we've presented before last year
at a workshop, and it basically combines
multiple Kinect cameras and displays into a single
coordinate system and by using multiple Kinect cameras
you can cover a larger room and create a larger
interactive space.
So, one sort of caveat with that is that given
the way the current Kinect system works you need a,
each Kinect requires its own PC and so the way
we do that with some of our larger installations
is to basically use a small PC like an Intel NUC
or something like that and you place that in the ceiling
and then you have a little bit of, a bit of server software
that basically transmits the depth and color images
to another PC which assembles the final image
or the final rendering and processing.
So, that's what this looks like here.
We don't need any audio, so yeah, that's good.
So this is what it looks like right now
and you can see over here what I've done is basically
just rendering the full scene on the left
and then I'm gonna
cut the audio here, and then,
and then over here on the right I just popped up
the Windows Performance Monitor that shows that
the depth, or, sorry, that the,
when you transmit all of these images together
you're actually taxing out a one gigabit connection
for this eight camera setup.
The cameras are in the ceiling, and then you'll notice
also in this box here that the actual frame rates
are actually not quite full 30 hertz, right?
So we're actually dropping frames in this
eight camera setup.
So that's bad, we don't want that.
So, let's look a little bit at
actually the Kinect depth image.
You guys have seen these before,
but it's helpful to look at them at,
you look at the numbers.
They're 512 by 424.
They're 16 bit but the actual depth values run
from 80 millimeters to eight meters.
That's 13 bits.
And typically when you get the depth image
from the Kinect SDK a zero value in that image
indicates that there was not a valid depth value
at that pixel.
And then one way to look at those images is that
they're actually alternating runs of zeroes
and non zero values and that's something that
will be important later when we talk about
the actual compression technique.
So what's the bandwidth if you were to just
stream those depth images from the Kinect camera server
to a PC?
So that's 424 kilobytes per image
and that's about 104 megabits per section.
So if you were to transmit both the compressed JPEG
color image along with the depth that's a,
clocks in at around 134 megabits per second,
and that's, you know, it depends a little bit
on your JPEG compression settings,
but these are all for the moderate settings.
So in theory with a one gigabit network,
a local network, you know, all your machines
are plugged into the same switch.
You can support seven cameras,
with the caveat that each of the depth images
have a worst case latency of about 33 milliseconds
because you're, you've tapped out your network connection.
So that's some pretty serious latency.
So, yeah, compressing the depth image, you know,
may allow for more than seven cameras
and also leave a little bandwidth open
for other payloads such as audio,
and reduce the latency due to network transmission.
So why am I focusing on lossless compression here?
You know, we could think about other kinds of techniques
using loss-y techniques, but I think if you,
if you actually were to apply say a JPEG algorithm
or some other loss-y techniques you would find immediately
that those techniques tend to introduce artifacts.
Now, the artifacts themselves are really quite damaging
around the contours of objects, and those,
these transitions in particular from, you know,
a very near object to a very far object
or even the worst case would be transition from
a non valid depth pixel such as the zero values
I discussed earlier to a non zero value
and so these loss-y techniques tend to
corrupt those things and they tend to have
a rather large effect when you think about
the depth image as a geometric interpretation of the world.
And in my, in my travels I have not found a commonly
available loss-y technique that actually does
the right thing around depth, around contours in particular
and so this is kind of a problem for us.
Now there are people who are working on these kinds
of things and in fact, you know, one of the things
you can think about is well just take an existing
loss-y compression technique like H.264
or something like that and, well,
most of these are optimized, you know,
around color images and don't typically support
13 bit or 16 bit per pixel images.
There are some exceptions.
One thing you might be tempted to do is to take,
you know, some low order bits of the depth value
and stick them in say the red channel
and then the remaining bits in the green channel.
That doesn't work because if you're in a loss-y
compression domain a single little error
in those high order bits are, yes,
the high order bits of that depth image,
say in the red channel, then, you know,
that is magnified because now that's a very large change
in the actual range value.
And so you can try that.
It doesn't really work.
And then there are some things out there like HEVC
version two supports 16 bit monochrome images
but it doesn't really handle the depth discontinuities.
So, and again, latency is really the enemy in all of this
and what I really am concerned about is end to end latency
where you compress at each camera host
and then decompress all in a single rendering PC.
And so compression and decompression speed
are both important, and there are a lot of techniques
out there that really focus on decompression speed
because often you compress an image once, store it to disk,
and so you can take all, typically,
you can take a lot more time in compression,
and that's not necessarily import,
what we want here, because we're looking at
end to end latency.
So this is a new technique.
It's kind of something that I came up with myself,
just sort of in messing around and sort of looking
at what people are doing out there.
I call it the RVL technique.
It's run length variable length compression.
So each successive pair of zero and non zero runs
is encoded in the order as the number of zeroes,
the number of non zeroes, and then the delta values
from pixel to pixel considering only successive
non zero values, okay?
So as I mentioned earlier you can view depth images
as alternating runs of zeroes and non zeros
and that's essentially what we're doing.
We're treating the zeroes very separately
from the non zero values by encoding the length
of the runs of each.
So, and then all of these values including the number
of zeroes, the number of non zeroes and these deltas
are encoded using what's called a base eight
variable length encoding scheme,
and I'll explain what that means.
So a variable length encoding, basically,
in this case what we do is we break the number
into as many three bit chunks as needed
to represent the number and then,
so, in this example here we break down
the 16 bit value of 42.
We take the low order three bits.
So that's the first chunk, and then the next
three bits as the last chunk, and then we used,
in this case, in this example, the first bit there
just is a continuation bit.
So it's set to be one if that's the last chunk required
and the key here is that smaller values require smaller,
require fewer bits, right?
So regions of the image with gradual changes
like a continuous wall or a surface, you know,
a continuous surface, give small delta values
and when they're encoded with this variable length
encoding scheme they're stored compactly
and then you can consider runs of zeroes
independently from runs of non zeroes,
these large delta values we might otherwise have
going from zero let's say to 4,000 are avoided, right?
And so that's sort of the trick.
And so another sort of thing you have to sort of watch out
for is that the variable length encoding only works
on positive values and so there's this well known
technique called zig zag encoding which takes care of that.
It can be implemented very efficiently in C
as these instructions without branching.
I'm gonna skip through this evaluation.
So the comparison techniques are our technique.
JPEG LS is a lossless version of JPEG
that really wasn't adopted too much.
JPEG XR is one that's built into Windows.
RLGR is something that I cover a little bit in the paper.
It's a Microsoft research paper that describes
the technique.
It's probably the most similar to this technique
and PNG often has implementation
that will have support for 16 bit per pixel images.
I add a couple of filtering conditions
just to see what happens when you smooth the image
using bilateral filtering and then
also just simple quantization.
You start throwing away bits and see what happens,
and then these are the test images.
They're videos.
It's a video of a real scene of about 1,000 images.
So this is the compression ratio you get.
So RVL is the first here on the left and the,
oh, I neglected a legend for these colors
but these are the different filtering conditions.
So the compression technique approaches about four
in the best case.
RVL stands quite nicely against XR and PNG
and the other ones.
The one that really matters here though
is the compression time.
Big win here for the proposed technique,
just the simplicity of it I think,
and I discussed the intuition.
I'm running out of time already a little bit here,
but decompression time, also very good.
PNG is holding its own and yet if you compare PNG
and compression time it's actually not.
It's dead last and second place in decompression time.
So that's kind of interesting.
This is, so going back to that video again,
if I just, in a second here I'm just gonna enable
the compression and you can,
we can watch the bandwidth go down to about 500 megabits.
So that's eight cameras both color and depth
and then our frame rates come back up to 30 hertz,
pretty solid 30 hertz.
So, the intuition here is compression,
in the case of compression, yeah,
having special knowledge of how depth images
look, you know, certainly helps in this case,
and then in particular handling runs of invalid depth
avoids these large delta values.
Very simple observation really,
and then I think the speed wins probably come from
the fact that in our case we're not manipulating
values at the single bit level as you might have
with say Golomb-Rice encoding, which is one
of the competing techniques, and we're not doing
Huffman or Lempel-Ziv encoding which is what
TIFF and PNG and JPEG LS do as sort of a final step.
So, if you do, perform a histogram over the number
of nibbles or four bit chunks that are required
across say a given image you'll notice that
yeah you only need one nibble for, you know,
a vast majority of the pixels and in particular
as you would expect if you apply a little bit more
filtering that's skewed even further in the direction
of just single byte, or single nibble chunks
and so this four X compression is obtained really
when you only use one nibble per pixel.
And finally just a, to think a little bit about
the end to end latency complexity
you can think about it, the big O notation for that,
and then you notice really quickly in this particular
configuration of multiple machines going into
a single machine that, you know, the compression time
C here is basically, you know, is because that's running
on all of the little PCs and fully paralyzed.
So that's sort of a factor that's a constant factor.
It's a dependent on the number of cameras
and the network transmission time is dependent
on the number of cameras and the compression ratio
because if you compress more you transmit less
and then the decompression time is then, you know,
also a function of the number of cameras,
but then also is, you know, can be paralyzed
on the machine with the number of cores.
So the temporal coherence would be sort of the next step.
I attempted a few things in my own experiments
and have not actually found a combination that makes sense.
I go about, into this a little bit in the paper.
And then finally, I'd like to say that in the spirit
of the one dollar gesture recognizer I'm providing
a 100 line, lines of C code, MIT license,
and that's in the paper, and that's gonna be included
in a future update.
I originally started writing pseudo code for this
and started putting it in the paper
and I realized that if anybody really wanted
to use this they would probably just translate that
back to C.
So I scrapped that and just included C in the paper.
So it's a little unusual as a academic paper
to see C code, but there it is.
It fits in on almost a column in the paper
and you guys are free to use it.
Thanks.
(audience applauding)
- [Man] We have time for one quick question, so, to you.
(audience member speaking off mic)
So.
- Go for it.
- [Diego] Diego Martinez from Sussex.
I was just saying, so you are,
you could use it, or it would be quite easy
in the state that you have it to use it
for surface identification 'cause you're using
the delta from one pixel to another like walls.
- Yeah.
- [Diego] Walls, they always have a constant delta,
and moreover as you have several cameras
from one camera you only have the derivative
in the horizontal, but that derivative
in the horizontal from this Kinect is different
than the horizontal from the other.
So you when you combine it you have something
that helps a lot already for surface identification
and moreover in your case that you have several cameras.
I don't know, it just came to my mind,
and I thought it was kind of interesting.
- There's a lot of interesting things you can do
with multiple cameras.
We're actually, oh, I'm working on
some little variant of the RoomAlive Toolkit
that takes into account some of these things,
mostly for filtering.
Not necessarily the compression, but yes,
and I think actually one of the things
that you would do, would sort of the next,
one of the things that you could do next would be
to think about the scene as a collection of planes
or, you know, or separable things for which
this linear prediction makes sense,
and you know, that's, that would be sort of a common
thing to do.
And then sharing those things across cameras of course
would be even better.
Thanks.
- [Diego] It was really cool, thanks.
- [Man] Okay, so, thank you very much, again.
(audience applauding)