- Good afternoon, everyone.
My name is Toshiki Sato from Tokyo Institute of Technology,
and today I'm about to present
our flashlight interaction system
named MlioLight.
At first I want to present short video of this work.
In this research, we developed information about a system
using multiple flashlight-like devices called MlioLight.
Our system enables the user to display
multi-layered information on the real world
using multiple flashlight devices.
Flashlights is one of our user-friendly
portable lighting device.
It does not require any specific manipulation,
and that area can be lit only by aiming the device
to the desired direction.
These characteristics of flashlight
is called flashlight metaphor,
and has been used as real world-oriented
information search and presentation methods.
Flashlights also have been used for multi-user interactions.
In the multi-user or multi-flashlight scenario,
flashlights are pointed in the same direction
and naturally overlapped with each other.
Previous researchers have been utilizing
the element of multiple flashlights for information sharing,
multi-user communication, or collaborative work.
In previous research, in order
to project information in the world like a flashlight,
a portable projector has been used.
It is easy to overlay information
on anywhere, especially in most flashlights,
most user situation.
However, a small projector-based flashlight system
require 3D position or post-tracking system.
In addition, when users overlap multiple projector's image,
real-time image color or brightness correction are required.
So, in previous works, using portable projectors,
it is difficult to display multi-layered information
through three or more flashlight devices.
Another way to implement flashlight projection system,
there is a method using a fixed projector camera system.
Pro-cam based flashlight system
is composed of an IR flashlight,
an IR camera, and a visible light projector.
The IR light emitted from the IR flashlight
is captured by the IR camera
and is mixed with overlay images.
Mixed images is immediately projected on the surface
by visible light projector.
The advantage of pro-cam based system
is the projection image mix with another light texture.
However, this method has also have a disadvantage.
If the system tracks multiple light source spots,
each light spot should be tracked independently.
However, once light overlap or cross,
lights are mixed each other,
and the information of each light,
such as shape, position, size, or ID data, are lost.
And it is very difficult to separate this again.
Previous works has been challenged to separate mixed light.
Inter-glows can identify multiple flashlight,
multiple modulated light,
ID by using frequency analysis technique.
However, it is difficult to apply frequency analysis
to each peak set of the camera.
Another research, such as restive shadow,
utilizes different wavelengths of IR light
combined with different types of IR pass filter.
However, number of available light is limited,
because it is difficult to increase the type of filters.
So, in this research, we propose
a projector-camera-based image overlay system
that supports more spare flashlights
called the MlioLight.
MlioLight system can separate multiple flashlight spots
and project different information on each flashlight
by using radio-synchronized image processing
and real-time image synthesizing technique.
The first advantage of MlioLight is
to increase the maximum number of users and flashlights.
This make it possible to use flashlight metaphor
in wider projection space.
The second advantage is the system can increase
the number of combination of information,
and also increase the number of layers
of overlay information.
This make it possible for users
rousing multi-layered contents
to selectively combine and overlay
information of specific layers.
Next, I'm going to explain prototype implementation
of MlioLight system.
Prototype system is composed of HD Projector,
grayscale high-speed camera with IR pass filter,
a PC, and a radio transmitter.
The flashlight device is composed of IR LEDs, batteries,
and a small microcontroller with a radio receiver.
There are no additional input buttons,
except for a switch on the flashlight.
On or off state of all flashlights
are remotely controlled by a wireless signal
transmitted every frame from projector-camera system.
Next, I'm going to explain flashlight separation algorithms.
As I mentioned, MlioLight system
broadcasts 2.4 gigahertz radio signal
that contains on or off signal of flashlights
before every shutter timing of the camera.
Immediately after receiving the signal,
flashlights emit for few milliseconds,
in this implementation, one millisecond,
according to the exposure time of the camera.
At the same time, the light emits
the system trigger shutter of that camera.
In this implementation, we developed
two different types of algorithms to control flashlight.
First mode is sequential flashlight switching mode.
In this mode, the system turns on
single flashlight per frame.
By turning on only one flashlight per frame,
the system can detect multiple flashlights separately
if they are overlapping.
The advantage of this method, it's simple and easy
to implement for higher frame-rate camera.
But the disadvantage is the scanning rate of each flashlight
is divided by the number of active flashlights.
Next mode is distance-based flashlight separation mode.
In this mode, the system turn on
multiple flashlights simultaneously
if the distance between lights are far apart.
I'll show in this figure,
the distance is longer than the radius value, r1 plus r2.
The system turn on two flashlight simultaneously
and separate them by image processing.
In this image processing, the system analyze image
with pre-captured background luminous threshold,
and separate each light spot by ray beam process.
Uh, like this.
In this prototype implementation,
the IR camera can capture visual resolution image,
at five...
150 frames per second in external shutter trigger mode.
The time of capturing single frame is about 7 millisecond.
In these prototype configuration,
the system support six flashlights simultaneously
up to 25 hertz for each flashlight.
Separated light image of each flashlight
is as input data, or by fragment shader,
to synthesize multiple image
and calculate the final color of the projection image.
MlioLight system synthesize three different
types of image in fragment shader of GPU.
First, image is a background RGB texture
that is displayed on non-light-illuminated surface.
Second image is multiple RGB textures
for overlay information for each flashlight
that are displayed on light-illuminated area.
Third texture is the light photon image
captured by the IR camera
that is a blend factor of each overlay images.
In case of blending two image with single flashlight,
each pixel color of front and back texture
are blended linearly by the luminous body of IR image.
When synthesizing multiple image with multiple flashlights,
final pixel color are calculated
according to the luminous balance
of each flashlight's image.
In order to display another image
on overlap area between two flashlights,
the system creates overlapped area pattern
from two separated flashlight images.
The overlapped area is used as a new flashlight image,
in this case, in this light, light C,
to synthesize specific image.
Next, I'll explain prototype application using MlioLight.
First, application is overlay information viewer
using flashlights.
Users can overlay additional information
on the projected image like match grains.
Second application is multi-layered information viewer.
Users can browse multi-layered contents by flashlights.
In these applications, each flashlights
have a different information,
and the user can change a layer
or the filter by changing the flashlight.
The users also blend multiple information
by overlapping multiple flashlight spots.
We also developed multiplayer entertainment application
for large projection mapping environment.
In this system, the system compose
of two sets of projector-camera system
and large three-dimensional projection target.
First game is a mystery finding game.
Using flashlight overlapping, the players
try to find secret number to open treasure box
by using multiple flashlights.
The last hint is displayed only when the player
overlap their flashlights each other.
Second game is a multiplayer 3D action game
using special game controller with embedded flashlight.
In this game, game object except player characters
are displayed only when the players
illuminate them with a flashlight.
The players explore the cave
while being helped by the surrounding support players
that have support flashlight devices.
We exhibit this game for three days last week for children.
Without explanation, the players realized
that they could overlap flashlights
and display hidden information.
Discussion.
First, I'm going to discuss the limitation
in image processing.
Capturing light spot pattern of a flashlight
is very important to natural flashlight pattern
to the overall image.
However, if multiple flashlights
concentrate in one place,
the camera image of overlapped data is easy to white out
because the dynamic range of the camera
is narrower than the human eye.
In order to address this problem,
the system should capture multiple flashlight image
with different exposure setting
and mix image to extend dynamic range.
Next is about image overlay missing overlapped area.
In this prototype, the system blended
multiple flashlight image in pixel unit.
The system also can determine
hierarchical relationship between flashlights
to blend multiple flashlight information another way.
Flashlight features such as size,
or total luminous value of each flashlight,
can be used to determine which layer is the upper layer.
Next, I'm going to discuss the future.
Number of images per flashlight.
In the current MlioLight system,
only one information is allocated to one flashlight.
In this configuration,
if the user want to change overlay information,
the user only has to change the flashlight.
However, if the number overlay information increases,
the number of flashlight also increases.
For multiple information on single flashlight situation,
we have to combine with extra input method
on the flashlight devices.
We also want to browse many-layered information,
such as time-lapse image or MRI data, in MlioLight system.
In order to browse this type of contents
with multiple flashlight devices,
we have to consider layer selection method.
In addition, in order to browse 3D volumetric data,
also need 3D cutting plane definition,
or 3D data estimation from flashlight devices.
We would like to investigate these methods
and implement in the future.
This is a summary slide.
Thank you very much for paying attention.
(audience applauds)
- [Man] Any questions?
If you have any questions,
please come up to the microphone
and speak up your name and affiliation.
So let me start the questions.
So you explain about
the distance between two flashlight positions
to determine the separation of the timings.
So what happens, how to calculate,
for example in this case, the surface is not planar.
Well, it's planar, but it is three-dimensional shape.
Then how to calculate the distance?
- So, after capturing the camera image,
so image processing system calculate image binarization,
and the leveling process of each light spot.
And calculate the center of the area, size of the area.
And the data is sent to the rendering system
in this implementation unity.
The laser is used for rendering.
- [Man] But on the camera image plane,
the distance is defined on the camera image plane.
But if the surface is non-planar surface,
then probably the distance between...
On the camera plane is sometimes
not working for such cases, or?
- Yes, so, we have to consider
the real distance in real space.
So...
This is my future work.
- [Man] Okay, thank you.
More questions?
So how many flashlights can you use in practice?
In particular, in the current system?
- So in the current system,
we have six flashlight turned on
simultaneously and overlap each other.
And the frame rate is 25 frames per second.
It's, the maximum number of flashlight
is depending on the frame rate of the camera.
So in this implementation,
we use that camera resolution frame rate,
so the maximum light is six.
- [Man] Okay.
If no questions, then thank you very much.