- [Audience Member] This is gonna be presented by
Andreas Fender and this is co-offered with...
by David Lindlbauer, Philipp Herholz,
Mark Alexa, Jörg Müller,
that's TU Berlin.
Take it away.
- Alright. Thank you Andy.
The work I'd like to present today is called HeatSpace.
My name is Andreas Fender
and the co-authors of that work...
Yeah, and you said them already.
David Lindlbauer, Philipp Herholz, Mark Alexa from TU Berlin
and my supervisor, Jörg Müller.
So the problem we're trying to explore
with this is the fact that information displays
are often placed by intuition,
so if we're looking at that image for example,
of a cafeteria's entrance area,
we can see a display in the upper right corner
which is visible in principle, you can see it,
but the people on that space are mostly
busy with picking up their trays and cutlery and so on.
So they're not necessarily looking at that display
and so it's important that the installer actually knows
how this space is used in the end
and that's often not the case
and that's also true for other environments like
in a manufacturing domain or in office spaces.
So the idea we have is that we are...
That we let uninstrumented users interact
in the physical environment
during the normal everyday routine.
And while they're doing so,
we measure their viewing behavior
and in the end, our system automatically proposes
display placement based on this measurements.
So this is what it looks like, so first;
we capture the environment and we do some 3D reconstruction
and then we let users interact in that environment
and we measure the visibility while they're doing so
and we reconstructing the physical environment based on
static surface and
moving objects.
We also put an application where installers
can try out potential display positions
and get written feedback
and finally; we optimize the placement of this displays.
So what we're contributing with is
an empirical analysis of user viewing behavior
and then optimize it for position and size of displays
and the overall goal is to replace guess work
with quantitative measurements.
So these are the components of HeatSpace
and I like to go to each of these in detail
so let's start with capture.
So before getting into the details
I'd like to show the example setup that we used.
So we were using three depth cameras;
Kinect two devices,
and they were all positioned so that
they were seeing the scene from an orthogonal angle.
The area of the space is 4 meters by 4 meters
and we used various different furniture arrangements
for the different examples.
So, capturing is pretty straightforward.
So we're using the feed of the depth cameras
to reconstruct
the physical surfaces
and then we generate real time meshes
based on these depth cameras
and they serve as a point cloud
and also for, as colliders for raycasts
which is gonna be important later on,
and we're also using the built-in skeleton tracking
that connect as decays built-in skeleton tracking
for user viewpoints.
So the second and one of the core components of the systems,
of course the generation of the heat maps.
And first of all, I want to mention
how we actually store the data.
So we're using a voxel grids
which are just uniformly
distributed in space
and each of these voxels
are storing local data about difference...
about different aspects
which I'm gonna explain in detail right now.
So, each voxel is storing,
first of all the persistence,
the distance, and the visibility.
Let's start with persistence;
so persistence means
how static an object is, we can see for example in that,
little video that,
the whiteboard is pretty static at the table
and the reconstruction we can see
the reconstructed static surfaces
which have a high persistence
and then we have medium persistence objects
which are like moving objects
and this are indicated with these
white glows and that rendering.
And then we have a distance,
this one is pretty straightforward so we're simply measuring
for each voxel and space, we're measuring the distance
from the user's viewpoint to the voxel
and in that image we can see that the table
close to that stylized user is very
orange which means that it's closed
and the wall and the background is a bit farther away
and therefore; it's green.
So visibility is a bit more sophisticated
and kind of most important and measurement we do.
So, the visibility is just a measure of the quality
of that voxel like how well visible is it for a user.
And we are splitting it into three components:
the first one being occlusion,
then the field of view, and the viewing angle.
So occlusion simply means for that instance and time,
is the voxel actually occluded or not is visible from
the users point of view and if there's a physical surface
in the way, in the line of sight,
then the occlusion is zero
or the value is zero
and otherwise it's one.
So the field of view is decreasing
whenever the angle between,
the users view direction
and the line of sight between the users hat
and the voxel, which is depicted as l is increasing.
So, we're getting the users view directions simply by
using the face tracking capabilities
of the connectors decay.
So that just means the further in the peripheral,
the voxel is, the worst the quality gets for that voxel.
And viewing angle is taking the surface itself into account.
So let's say there is some geometry within that voxel
and that geometry also has a surface normal
and then we are simply measuring the angle
between the line of sights and that's surface's normal
to get the viewing angle, so what that means is that
when you are looking at a table for example
with a very narrow angle,
then the quality of that measure is fully bad
and if we're looking straight at the wall for example
then the value for the viewing angle is very good.
So in the end, we're multiplying all these things
to get our visibility.
All right, so all these measurements are
calculated at each moment in time for each frame.
What we do in the end is we have time accumulating them.
So the persistence and the distance and the visibility;
each are each time accumulated.
So for example, this is the visibility for
the first instance in time where we can see pretty much
one focus point but over time it's getting
a bit more distributed and over time we getting...
We are finding out which areas are getting more attention
which are getting less attention
and I'd like to talk about one of the implications
in particular about occlusion, so for example,
here we see a user interacting at a table
and for that instance and that time we can see on the table
there's occlusions either zero or one, depending on...
If it's occluded by the arms or not
but over time we can see that
the whole area is kind of bad quality
even thou they're not occluded in that instance in time
so that means that this place will be positioned
further away from the user, even though the viewing angle...
Everything is nice but still since the hands are always
waving in front of them, of the user...
Still look back, surface area; for example.
All right, so the last big component is
the display placement.
So the way our application is structured like that.
So first the installer needs to specify some displays,
so there's some parameters,
some of them are optional,
some are required.
And so, the first parameter is the number of displays
so that's a value that the installer has to put in.
And also, for each display the resolution has to be set
and optionally, the installer can
kind of try out some initial positions for the displays
and optionally the size can be fixed
or can also be delegated to the optimizer
to optimize the size of the display
if that's a degree of freedom for the installation.
And in the end, based on these values
we want to find optimal display surfaces.
This is what the application looks like,
so we can explore the reconstruction
and the insular can just freely position
some potential displays anywhere in the room
and then set some parameters
and move the displays around
and while the installer's doing so...
Here she's getting real-time feedback
so within the displays we can see different shadings
indicating how good that portion of the display can be seen
and also the quality is being, output.
So in this place can be positioned
at any static flat surface
which can be vertical or horizontal,
and if it's horizontal then the aspect can also be rotated.
I'm finally after the installer
has set up some initial positions.
The optimization can be triggered
simply by clicking optimize.
All right, and I want to briefly
talk about the optimization algorithm.
I don't have time to really go to detail
but I just want to provide you an idea about
how we are actually doing the optimization.
The first step is to find planar
and geometrically persistent candidate regions.
And the way we do that is...
we are creating a connected components graph
and that graph is based on persistence and planarity.
That means for example if we're having a flat table
and a white board which are like close to each other
then these two are different connected components
even thou they are physically attached.
They still have a different surface
no one there for their true different connective components.
The second step is to sample potential display positions
and then for each display to be optimized,
we do the following, we place in the display
greedily on the sample with the highest score
where there is no display already.
That step is kind of important.
If displays end up connected on the same connected component
we using alternating gradient descent approach.
This means for example if there's a big wall
or a big white board, whatever
and two displays are positioned
on that big connected component
then it will be kind of unfair for one of the displays
that we first optimize the first display
and then the second one has to figure out
some other space; somewhere close.
What we're doing instead is putting them both
at the same time and that big connected component
and then, gradually, locally optimizing
both of them at the same time.
That's the alternating gradient descent approach
but of of course, way more details in the paper.
Also, I'd like to show some example results.
In that first example we are having three users
sitting on a big table
and what we can see here is that
the second user is blocking the view, to the wall.
For the first user, which is also reflected
on the real timeline feedback
and of course, the optimizer is taken that into account
and moving the display so that it's not...
So that the occlusion is kinda minimized.
We can also see two smaller displays to the right
and even though they are in kind of in the peripheral
of the first user, the resolution requirement
in the size requirements forces these displays
to be really close to the user.
The second example are two users
sitting opposite each other
and what we can see here is that
the first user has a pretty good overview
of the room behind the other user.
However, if we are looking at the combined heat maps
of both users at the same time,
then we can see that only in the peripheral for both users
which is like the wall to the right
can actually be used for displays
and the optimizer is of course taken that into account
and position the displays so that they are visible
for both users, not only for one of the users.
Of course we're having some limitations,
some of which we also try to take on the future.
First limitation is that displays
have to be attached to static surfaces.
What we mean by that is that we...
For example we can not use ceiling suspended displays
or displays which are just standing
in the middle of the room
because if we would use these displays
then they would change the whole physical layer of the room
and all the occlusion patterns
and everything would just look different
and therefore, we can not optimize these.
And also, of course, there might be a...
There might be a behavioral change after installation
and this is especially true for interactive touch displays.
Which means, displays which...
where the intention is to actually change the interaction
of the workflow within the space
and therefore, we are more focusing on displays
where the intention is to not change
the workflow too much to seamlessly
integrate this displays into the workflow,
primarily, information displays
or otherwise we can just keep measuring,
so after an initial optimization we can just keep measuring
and maybe refine or verify the display placement.
To conclude, the system I just presented is called HeatSpace
and we're letting uninstrumented users
interact in the environment
and while they doing so
we are measuring persistence and distance and visibility,
and this work are stored in voxel grids.
Measurements are time accumulated,
so they're measured over time
and the display placement optimization
is based on that empirical data,
and the overall objective is to replace
guess work with quantitative measurements.
All right. And with that I like to end the talk
and of course, I'm open for questions, thank you.
(audience applause)
- [Audience Member] Steve Finder, Columbia University.
I love work that tries to automate
the performance of certain things that people
would normally do by hand.
I have a question and it's something I encounter
and many other people encounter as well.
You have a multi variated equation there
that's going to determine the score for something,
so what made you decide on the particular weights
of the individual things you were combining together
and did you do anything to try to validate
the right way to do things?
- Yeah.
So this is kind of briefly,
also, as discussed in the papers...
That's a very good question.
Also something we thought about.
This different weights, of course,
right now, they have to be set by the installer
and that would require some knowledge
and maybe some intuition about
how to weigh occlusion against peripheral field of view
and some depends on, like for example
if the content is moving a lot then
it's maybe not so important in the sample field of view
because if there is motion then we can kinda relax
that condition about the peripheral view
and then the installer can not adjust the exponents
for that value but of course in the future would be
kinda cool to also explore setting this parameters
automatically based on the content of the displays.
For example if we know that the content is animated
then we can adjust the weight for the peripheral view
or if we know that the display has a really bad
viewing angle range... - All right.
- I don't know the exact word for that right now,
but like these old displays which are really hard
to see from the side is not such a big problem
nowadays anymore, but if we know that
and of course the viewing angle would have more weights
so to say, then that display
has to be seen from the front.
For example...
So there's a lot to explore in that direction also.
- [Audience Member] And also doing studies, you see,
with a particular layout, how people actually perform
to get the sense of which things really matter.
One thing I noticed that didn't seem to be in there
unless I missed it was if you and I are
opposite sides of the table
and there's a display on the surface of the table
and it's got text that we have to read,
one of us may end up reading it upside down.
Unless, you consider that both of us read it on the side,
so, did you try to take that into account?
The orientation of the content?
- Actually, not yet but that's a pretty cool idea actually.
Yeah. So I guess we only have that problem with
horizontal displays, maybe that's something where we
need to investigate a bit more in detail
how it works differently when
we're kinda focused on horizontal displays,
when all these additional challenges
have to be taken into account.
Yeah. - [Audience Member] Thank you.
- [Host] Any further questions?