Hi everyone! My name is Virag Varga, and I'm here to present on behalf of my co-authors:
Gergely Vakulya, Benjamin Burgisser, Nathan Riopelle, Fabio Zund, Robert Sumner, Thomas Gross,
and Alanson Sample. Our work is called Real-Time Capture
of Holistic Tangible Interactions. This work is done in collaboration between ETH Zurich,
Disney Research Studios, and University of Michigan.
When we plan to blend the physical and virtual worlds we often focus on how
to get the display right; for example, on how to identify object markers or a horizontal surface
so that we can realistically render a virtual overlay. While this experience can be already
entertaining and engaging, these applications often focus on a very narrow view of the world,
and miss the details of the rest of the user's physical environment, and her interactions in that
environment; such as touching various objects or being present at a certain location. This
overall interaction capture problem can easily become even more complex once there is occlusion,
or when there are similarly looking objects present. In our work, we wanted to find a capture
solution for holistic tangible interactions, which could single-handedly cover all the scenarios from
the previous slide. Ultimately, we wanted to find a capturing technique, which could tell us who
is doing exactly what and where. So that we could incorporate these events into the storytelling of
the application for a better, a more immersive experience. The ultimate question is of course,
what technology would be best suited for such a holistic capture. While there exist various options
with different advantages and disadvantages, our proposal is to utilize body channel communication.
BCC uses an electric signal confined to the user's body to transmit digital data along the skin.
In the tangible interaction capture scenario, this would mean that tangibles could invisibly
broadcast their IDs to anyone in direct contact with them. The TangibleID system we propose here
looks like the following: in the augmented reality setup the application starts as normal. The AR
camera scans for markers, and upon finding them, an overlay is displayed. The difference comes
once we start interacting with the objects. We know immediately if the user picks up something, and we
also know exactly which object that is. Once both are in hand, we can trigger for a special event.
The TangibleID ecosystem consists of the following elements:
an instrumented object, which can be anything like a book, or a bottle, or even a lantern,
and then of course, we have the instrumented user himself. Once the user comes into contact
with any of these objects, the touch event is recognized. By the way, it's not just various
objects that the system supports: locations can have their unique identifier as well
and even other users. The TangibleID system would support any interaction between these types. Once
we extend the system with a display, the event is then forwarded through a bridging mechanism to the
application level. And last but not least, in our system we added an optional developer extension
an Overwatch application, which can oversee any network activity from a computer.
Body channel communication had been around for a while. Our group published the TouchCom system
some years back. However, the purpose of such system was to optimize for generic data streaming,
making sure that longer messages also arrive intact between the transmitters and the receivers.
In TangibleID, however, the network's purpose is different. Here we want to send short
messages to indicate an ID that's picked up through touch. Several questions arise:
What should be the packet format? What does error correction even mean, and how should
it be applied? What beaconing peace can be used, and what does it mean with regard to
touch recognition performance? And last: how can we optimize the system for simultaneous touches?
Overall, while building TangibleID, we had to address several questions:
first, as just mentioned, the network protocol had to be tailored to touch recognition.
This meant optimizing for reduced latency even while keeping the identification accuracy high.
Moreover, the data forwarding had to be addressed. A hardware iteration was needed to reduce the
physical size, so that we can integrate the transceivers into various everyday objects.
And last, a headset and tablet integration needed to be considered as well. Now we don't have time
to go into the details of all these points, but I quickly want to show some data on how we evaluated
the performance. Here I'm going to discuss specifically the simultaneous touches scenario.
This plot shows four different beaconing strategies that we evaluated, each with a different transmission
frequency. We had two transmitter objects continuously beaconing their IDs, while the user's
wristband was set to receiving mode. We can see on the plot that having a slower transmission rate
- this can be seen on the RIGHT hand-side with 15 to 50 millisecond wait-time between beacons -
this resulted in more correctly received messages. This case yielded 75 percent correct transmission,
while with the short wait-time between the beacons this performance drops to 43 to 44 percent.
However, when we translate this network performance
to a touch recognition metric, then we see that using a one second evaluation window, we are
actually guaranteed to receive more messages with the more lossy transmission protocol.
The higher number of messages are advantages because then we can be more certain that
the touch event actually occurred during the evaluation window - as a sort of error correction.
If now we want to lower the latency, then we can also take a look what result we would get using
a different evaluation window. In this case, we can see that even if the touches are evaluated
every 50 millisecond, we would still roughly get two separate ID messages per transmitter.
This should be sufficient for accurate immediate touch detection.
Once we built TangibleID, the next question is what interaction modalities does it enable.
We identified the following ones: as a base case, we can achieve context awareness by being able
to recognize objects the user is dealing with. This can be extended to a group of dealing with
so-called smart objects either by saving state information or interaction history
directly to the object, or by the object understanding the presence of other objects.
Next, we can have interactions anchored to certain locations:
either by the presence of a given user, or by a given object at a certain location. Last, we can
also recognize multi-user interactions, such as several users interacting with the same object,
or the users interacting with each other - like a handshake or high-five.
To demonstrate the potential of TangibleID, and these interactions,
we created a complete showcase set in the Haunted Castle. Here I'm going to show only a few examples,
but please look at the paper, or the short video demonstration for the rest of the scenes.
In this first example, a ghost is floating around, and two physical objects are lying on a desk:
a lantern, and a book. The objects are identified instantly as the user touches them. Depending
on which object the user picks up. it kills. or summons back the ghost. This example shows how
context awareness can be achieved through TangibleID. Events can be anchored to certain locations.
Here the player sees a painting of a lady. If the user brings either the lantern, or the ghost
book close to the painting, and only then, the painting suddenly comes to life. The floor tile in
front of the painting can recognize the objects brought there so the displayed AR content can
automatically adapt even if the interaction is not in the line of sight of the AR camera.
In this last example, we show how we can support novel interactions where several users play
together. Users can unlock the floor plan, and all extra information about it by interacting
with a map. If the user tries to unlock the map content alone, they will only see part of it.
All users are needed to be present to see the full map.
The map is a touch sensitive object that listens to incoming user IDs.
In conclusion, in our work we introduced TangibleID, a holistic tangible interaction capture system.
We showed how a generic application workflow should look like, what software components need to
be taken care of, and how BCC can be redesigned for such touch recognition. And finally, we identified
seven different interaction modalities, which were then demonstrated through a complete showcase.
We hope our work can inspire future application developers and designers
to integrate more tangible interactions for a truly immersive experience.
Thank you for watching, and we are happy to address any questions.