- Thank you.
Good morning and thank you for coming.
I'm here to present Lightbee,
A Self-Levitating Light Field Display
For Hologrammatic Telepresence.
This was thesis work by Xujing Zhang,
who was unfortunately could not be here.
She is now working for Belecom.
We'd also like to acknowledge the work
of Shawn Braley and Calvin Rubens in this project.
I'll be presenting more of the background of this work.
One of the issues with VR helmets is that they are heavy,
cumbersome, and the field of view is rather small,
and poor resolution.
We also lose the ability to see the world and to be seen.
The occlusion of the head and eyes
make co-present face-to-face communication difficult,
and we don't capture the facial expressions
when teleconferencing.
There's a need for 3D displays
that live in the real physical world.
So what's going on here?
Well, nonverbal cues are actually richer
than you would think
and more difficult to convey.
Eye contact is crucial to multi-party conversations.
And we want to preserve the head orientation
and we need to render multiple perspectives
as well as allow for management of personal space.
According to Hall,
between 1.2 meters and 3.6 meters
is the area known as Social Space
and this is where Lightbee operates.
We need to move, or we need movement to convey proxemics.
However, robots may represent the head too small,
or they may not be fast enough to convey
those head movements accurately.
So now I'd like to highlight some previous systems who've,
which have tried to address some of these issues.
Here, an early system, we see Ari Paulos there with PRoP,
a series of telepresence robots
that include wheel robots and blimps.
The form factor of the wheeled robot is something
you'll see in various other examples later
where there's a screen mounted on a stick
and moves around on a wheeled robot.
Here's some commercial examples.
Ava 500 Cisco and iRobot.
And on the right, Double 2 from Double Robotics.
And in terms of controlling these systems,
this is typically done with touch screen.
However, there are some examples, as well,
where this is MeRobot
with Breazeal where they could capture some
of the richness of the facial expressions
and movement and translate that into expressive
movements with the robot.
Here's an example of tracking the head position
for remote viewing through a drone.
And then here we see a corollary
with a flying display in the drones.
There are also multi-projector systems.
So multi-view was a third party,
or a multiparty video conferencing system,
uses a retro-reflective screen
with three distinct view ports.
However, it did not provide serioscopy
and nor did it have continuous motion parallax.
Telehuman was then a system that allowed
for the remote person to be represented
by 3D volumetric video.
It relied on shutter glasses
and as such, was not multi-user.
Then we have Telesar, Telesar4 actually,
is a system and it used retro reflection
projected on a robot,
but the robot itself didn't move beside the arm.
Then we have Pan & Steed.
They presented an experimental setup
with a cylindrical retro reflective surface,
nine projectors capable
of projecting nine distinct view ports.
The problem with this is that nine cameras were needed
to capture these views.
So then Telehuman2,
which was presented at CHI last year,
put all of these elements together into a light field.
The light field displayed that rendered the full size human.
However, it was not mobile.
- [Rimmer] Has anybody seen my legs?
(laughter)
They don't appear to be below my waist
where I normally keep them.
- [Lister] Holly, what's happened to Rimmer's legs?
- [Cat] Here they are right here!
(laughter)
- [Rimmer] Stop them!
- [Lister] Come on, leggy's.
This way, over here.
- 592, that's where the hologram simulation sweep is.
- So that was a clip
from the British sci-fi show Red Dwarf,
if you recognized it.
The show features a flying hologram technology
called Lightbee.
So Lightbee is a small hologram projector
that projects hologramatic recreations of the dead,
and contains the personalities of the deceased crew members.
It buzzed around inside the projection like a bee,
and hence the name of the device.
And we named our system after that.
So the problem is we cannot project in mid-air.
But, if we fly a projection surface,
we can project multiple views on that surface.
By using lots of projectors spaced apart,
we can create many view ports that are retro-flected
by a very lightweight fabric that can fly
and thus, a flying light field is born.
Each eye thus picks up a different image
from a different projector no matter where you stand.
And that also means you got serioscopy
and motion parallax for free
as you move along the projectors.
So here's LightBee.
Now, I just want to say it looks awful,
but that's because the shutters of the 45 projectors
are interacting with the shutter of our camera
and we haven't really been able to synchronize that.
You don't see that when you actually experience the system,
but it's very difficult to film.
So here's the drone,
it represents the head of the remote interlocutor
as it projected on light field.
And that means the users can see the head
of the person flying around in 3D.
And it appears as if it's actually inside the drone.
Again, that's not something I can show here in 2D video.
So here we see a user interacting
with the remote participant
and the social paramics are actually quite natural.
Not only does it conserve eye contact,
but it modulates intimacy by moving forwards and backwards,
just like you do in a regular conversation.
Users can step forward, see head orientation,
or make eye contact and if you have multiple users,
they will see natural eye contact and head orientation
towards their location and all that is preserved.
So the system is multi-user by nature.
Alright, implementation,
so the drone is a cylindrical retro-flective surface,
which is mounted on a 38 centimeter diameter frame.
The drone has four brushless motors,
a Micro MultiWii flight controller board,
and a ESP-8266 WiFi board.
And the controller is a customized version
of MultiWii software that's modified to work
with the brushless motors
and with our customs communications protocol
because we need very, very low latencies,
like in order of ten milliseconds round trip kind of time.
Power's provided by 1300 milliamp power battery
that sustains approximately four minutes of flight,
'cause this thing is actually quite heavy.
And then a unique pattern of icon markers,
you see those little dots there.
They allow us to track the drone in real time.
Again that goes very, very fast,
very for low lengencies.
And all that is optimized
to be able to control the drone precisely
within the space of the projection system.
One real issue with this it that the cylinder
provides a lot of drag and so you don't want
this kind of messy, air pressure pattern
underneath a drone because you really want
to have a jet coming out.
And for that reason, we use special props
that are not as efficient but they are angled.
And so they push air down faster.
It means the drones flies a little bit shorter,
but it does mean it flies.
The retro-reflector fabric is very lightweight.
It allows the drone to reflect the projection
from a particular angle precisely into 1.3 degrees,
so that was engineered as such.
One part of that that's critical is the vertical diffuser.
It helps diffuse the light vertically.
But it limits the light being horizontally.
And we use an RPC photonics diffuser
with a 56 degrees vertical scattering angle.
And that allows the drone to move up and down
without a loss in brightness.
It also allows the projector rate
to be mounted above the heads of the user.
So then as they move along,
the brightness of the projection changes 50 percent
by degree, by degree
and so that's how you create the light field.
A smart projector ray is mounted
on a rail 45 PicoPro 72P Smart Projectors
with each an Odroid system that renders its perspective.
So this is a massively parallel computer
with 45 view port renders that render in real time,
45 images each one tenth of a second.
Since all the light from the projector is retroflected
in an narrow area,
that means the Pico Projections
are actually quite bright, even in broad daylight.
The surround video of the remote persons produced
using three stereoscopic cameras,
one per 20 degrees.
If you wanna do full 360, you need nine.
And that constructs the 3D volumetric video of the head.
Each ZED camera is connected
to a high end GTX1080 graphics card,
which then computes depth images and texture maps.
These are then sent over UDP
and then softer on Odroids receive
that to calculate their own offset for the projectors.
So these are the six images
that you would have to send over Skype.
And then a projector clusters of 15 projectors
will go in there and calculate the plus seven
or minus eight depending on where their location
is in the array
using a relief mapping algorithm.
Here's how the algorithm works.
Each pixel in the image which represent a light ray
coming from the back of the screen
towards the 3D objects in the screen.
And so it's bounded by the depth map which we see here
as on the top, A,
and at the bottom, B.
So we then tried intersect the depth map
with that light ray.
And so we see
that there's intersection here at number three,
which is incorrect,
because really what we want is the top intersections,
so we do some searching in the depth map until we find C,
which is the correct answer here.
And then we look up the color in the texture map.
And that becomes the color for that pixel.
Mark is on top of the LightBee drone.
Allow it's location be tracked
with eight Vicon motion cameras.
The wave points are set for remote head movements by Kinect.
So the Kinect measures the head movements
as the person is talking.
PID loops then control
and adjust the quad copters thrust along the X,Y,ZED axis.
And then two cameras on the drone allow
for the remote person to also see the scene on two displays.
That means the system's only one way, if present.
But that's not a fundamental issue.
It's just an issue of money.
We did a very limited user study.
Mainly because we didn't want to kill
our participants with this drone.
So six dapper participants took part in the study
and were asked to move around the gray area
in the image here.
And we asked them to engage in a triadic conversation
for about three minutes with an actor
who was represented on the drone.
And in each conversation this actor made sure
that whenever he was talking to the right participant,
he'd be looking at them,
left participant, looking at them,
so making sort of clear what his intentions were
with head orientations.
Then after each session,
The participants were asked
to reflect upon their experience.
Initial impressions were positive.
Most participants noted that the movements
of the LightBee helped them facilitate turn taking better.
P4 claimed that the movement
of the LightBee gave me a better idea
of what the remote person was doing.
And P6 claimed that the LightBee's movement
was easy to understand and smooth.
In terms of proxemics,
P1 claimed that he felt a kind of pressure
when it approached me.
And that's exactly the kind of thing that you'd want.
There's very few video conference systems
that are capable of giving you that eerie feeling,
that something's gonna happen when a person gets closer.
The drone definitely helps with that.
P6 also noted eye contact was really helpful,
like in a class or a meeting.
And P2 noted that it was convenient
to show the person around by moving herself
and that is was cool to look around the person
by moving around the LightBee.
Surprisingly, only two participants complained
about the noise.
The system currently is rather noisy
for an application like this.
All right, so some limitations.
Well currently, the system only runs
at ten frames per second.
That's mostly due to the Odroids.
They're just not fast enough for the system on chips day.
As they get faster, that'll improve.
It's not a huge problem.
Ghosting is visible.
That's due to imperfect retro-flection.
When you have multiple projectors,
retro-flecting on a surface,
you might see the other projectors slightly.
That can be solved by using a limited form of face tracking
that turns off the projectors where there's no face.
Bi-directionality of course.
So we'd love to have a system that's fully reciprocal.
And that could be done by placing ZED cameras
around the pod.
And of course I already mentioned the noisy propellors.
We've been looking into things like ultrasonic designs
that speed up the propellor so much
that the noise is actually no longer audible.
And of course there's anti-sound solutions,
or we could just change the whole design to use,
for example, gas propellants or something like that.
But that's something to look into for the future for sure.
All right, so concluding,
we presented LightBee,
a drone based hologrammatic video conferencing system
that uses a retroflective multi view projection system
to create a high resolution light field transmission
of 720P.
It supports multiple participants
without any need for glasses or headsets.
And it allows for moving proxemics,
as well as 3D facial expressions to be transmitted fully.
And with that, I'll take questions.
(applause)
- [Woman] Thank you very much,
I already see a hand up.
- [Man] Hi, I'm Claude Beonestrom at BM research.
First, I mean, congratulations.
Incredible amount of work there.
I can see I've done much less than that I know.
But one thing I was thinking is,
what the drone gives you that,
I mean given your installation,
you could be using a tethered arm
from above or wires this kind of stuff.
What you get and get more time, more flight time?
- Very good question.
Yes we could suspend this on a wire.
I think one of the reasons it's a drone is
because we've done a lot of drone's work.
This comes out of Bit Drones
and so we have the controls and we had all that.
But I think also there's a lot
of benefits to having drones.
First of all, the face appears inside the drone.
So the drone disappears 'cause it's just a retro-flector,
so the whole thing disappears.
And with a wire, you don't really get that sense.
So the moment you introduce an artifact
that's not real,
people just know that this is not real
and then it just pops the whole telepresence thing.
Drones are also very responsive in a way
that a tethered system would be difficult to build.
So there's a bunch of reasons.
And finally, if you fly out of the room,
you can actually go and inspect your building site.
For example, so you can have communication with someone
and then go off and fly off around and talk about what you--
- [Claude] But not the projection?
But not the projection?
- But the projection would not be visible then, no.
- [Woman] Hello, Ibass from media innovation lab in Israel.
So first, very, very cool. - Thank you.
- [Ibass] And I was wondering if you're participants
gave more general impressions, whether they liked it or not.
Did the compare it to regular video conversation,
just about experience in general
and not the details of the movement and understanding?
- Yeah, I'm not gonna make any grandiose claims
because literally we just had,
you know, people rushed in there, rushed out.
But it did seem that they were definitely
getting what this thing was doing.
And I think one of the really interesting elements
that I would love to investigate further is that the,
you know I've always said telepresence will never be a thing
until there's the threat of killing someone.
Right?
And proxemics is actually based on arm length.
Right?
Two arms length is the maximum distance.
One arm length is the minimum distance.
So this drone, seems to be amplifying,
not only does it convey it,
but it amplifies it because it is slightly dangerous
and people are aware of that.
So I thought that was really cool.
It'd be very nice to see what you could do
with that design
and see if you could dial up the social presence, you know?
But yeah, so people did like it.
They did kind of think it was like a natural system.
Thank you.
- [Man] My name is Yu Kata from Universal Sectics.
I have a question about the material
that is was already hectored.
So I saw that the problem was the ghost imaging
because maybe using the micro beads based on vector.
But there's also the quantity of base already hectored.
Is the base sharper? - Right.
- [Yutaka] It's much more precise projection.
Have you ever tried that?
Or have you considered to use that kind of material?
Also spray base the hector so maybe you don't have
to stick to specific material.
You can design for any safe water spray quality.
I just wonder,
is kind of the design place for the material?
- Yeah, that's another thing.
That's a wonderful question.
Thank you for the suggestions.
I think that's just something like I mean
there was a lot of work involved in getting the system
to work to begin with.
So, you know, we didn't look at sort of optimal materials.
We really did like the fabric because it was so lightweight
and it worked really well with the drone.
I will also say that there has to be
that combination with the diffuser.
Not just for the diffusing vertically,
but the diffuser also controls
the angle very much more precisely
than the micro beads would.
So there's a huge design space there for trying
to optimize the reflection,
so you don't get the ghosting
and, you know, so I agree with you.
Maybe we can talk after.
- [Yutaka] Thanks so much.
- Thank you.
- [Woman] Any other questions from the floor?
Just a quick reminder
that if you are sticking around for a few talks,
please move towards the center,
so that any one else sneaking in late can grab a seat.
We do have time for one more question,
which I do have.
I was interested to notice that you've got a patent
for what is the appropriate distance.
But does that take into consideration
that there can be cultural differences
between what is appropriate?
- Yeah, absolutely.
You know that's a good question.
I think that you really do need the reciprocal system,
like so both sides,
in order to be able to convey that.
So my intent is always been
that if you just mimic the cues that are,
or convey the cues that are coming in the natural world,
people will figure it out.
So we don't have to control for that.
But in our system,
because it's only,
because it's asymmetrical, you would still have issues,
because you know on the IMAX,
the faces aren't exactly correct
and the social barxamics don't work.
So anyways,
I think it's a very interesting area of research.
Even just investigating whether that does or does not work,
but I think it does.
- [Woman] Thank you.
- Thank you.