This is Yihui Yan from ShanghaiTech University. We present a new tangible user interface.
The interface consists of an ordinary screen and the MouseToken tangibles.
The MouseTokens are commercial optical mice or customized wireless tokens with optical mouse sensors inside.
Once the tangibles are put on the screen surface,
their locations and orientations can be tracked in real-time.
The user can use them to interact with the digital content.
The benefits of the interface are twofold.
First, it works with both touch and non-touch screens.
Unlike tangibles based on capacitive touchscreens,
it does not have the grounding issue,
the user does not have to keep seizing the tangible for interaction.
Second, it is cost-effective.
Its hardware requirements might have been met in many personal computers.
The customized tokens are also very cheap.
The setup is easy and incurs no hardware modification.
Next, we explain how it works.
The key component of the MouseToken is the optical mouse sensor.
The sensor is a high-rate and low-resolution camera.
It continuously captures images of the working surface
and senses the movement by comparing the difference of consecutive images.
Our customized token mimics the commercial mouse
and packs the sensor into a smaller case.
When putting the MouseToken on the screen,
its mouse sensor is able to capture the display content.
For example, when the screen displays white, the captured image is bright.
When it displays black, the captured image is dark.
The capability of the MouseToken allows us to track its location on the screen surface.
The mechanism can be explained with a simplified example.
We divide the screen into left and right regions.
We vary the light intensity of the display content differently in the two regions.
The left region varies faster than the right region.
When the MouseToken is placed on the screen,
the light intensity of the captured images varies correspondingly.
By counting the frequency of the intensity variation,
the region where the token locates can be inferred.
The example explains the basic idea of modulating display content to transmit location information.
To extend it to track tangibles in practice,
the screen surface is divided into much smaller regions to increase the precision.
Each location is represented by a unique intensity variation signal.
Hence, the position of the token is inferred according to the received light signals.
Further, we use a high-refresh-rate screen and spatial-domain patterns
to reduce the flickering issue of the modulated display content.
Next, we explain how to track the orientation of the token.
We notice that the mouse sensor can capture not only the intensity variations
but also the fine structure of the screen panel.
A screen panel is a huge array of RGB pixels.
The pixels are separated by unlighted boundaries, which are dark gaps in the captured images.
When the token is rotated, the gaps are rotated as well.
The orientation of the token can be determined
by measuring the direction of the gaps in the captured images.
We implement the tangible interface with a desktop PC and a 240Hz monitor.
We use several optical mice and customized tokens as tangibles.
The software is based on OpenGL.
It modifies the original display content to encode the location information.
We measured the tracking performance in practice.
The positioning accuracy is determined by the granularity at which the screen surface is divided.
Finer granularity implies longer signals and hence longer positioning delay.
We divide the screen into more than 1000 grids in the current setup,
which gives a precision of 1 square centimeter, similar to some existing tangible interfaces.
The error of orientation is almost within 2 degrees, which is very accurate.
The positioning delay is 0.41 s on average.
Because of the waiting time, the positioning method is integrated
with the mouse sensor’s built-in relative tracking to improve the smoothness.
Using it alone is more suited for interactions that can tolerate a slight delay.
Next, we show some applications of the prototype.
The first application is a puzzle board.
The user needs to select and drag the letter piece to fit in the correct hole.
A commercial optical mouse is used for the interaction.
Its large form factor allows for easy-grasping,
which is suitable for young children, whose fine motor skills are not fully developed.
The second application is Frog Balance.
The balance automatically “weighs” the frogs and masses on the plates.
It supports weighting multiple tokens simultaneously.
The balance is designed to help children to learn basic arithmetic concepts.
The third application is color widgets.
The tokens are used as knobs.
The rotation angle of the token is mapped to the value of the color component,
which allows for eye-free and fine-grained color tuning.
The fourth application is an interactive D&D battle map.
The map can automatically locate the characters.
The user selects the character actions by sliding the token towards the menu item.
Depending on the implementation, the token generates physical feedback to the user, such as light, sound, and vibration.
In this talk, we introduce a precise and unobtrusive method to track on-screen objects.
The location information is encoded into the variation of the display content.
The flickering of the variation is hidden by the visible light communication and high-refresh-rate display techniques.
We believe it renders an alternative and interesting approach for on-screen interactive designs.
Second, we apply the tracking method to design and implement a cost-effective tangible interface.
Its tangibles use optical mouse sensors for tracking.
This choice effectively reduces the hardware and setup cost of the system.
Its cost-effectiveness implies an affordable way to deploy and study tangibles in practice.