Hello, everyone. I'm Yuhan, I'm a fifth year PhD student from
Cornell University. Today I'll be talking about ShadowSense:
detecting human touch in a social robot using shadow image
classification.
Let's begin with some background. Just like
human-human interactions touch is an important modality when
interacting with robots. Despite that not so many social robots
have integrated touch as an interaction modality. This is
possibly due to that the existing robot tactile sensing
use force or capacitive sensor arrays, which require a large
number of sensors to achieve a useful spatial resolution. For
example, the huggable robot used more than 1000 sensors to
achieve full body detection, a child robot CB2 uses nearly 200
sensors. Another shortcoming is that force sensitive arrays
require relatively flat and stiff surface and are
insensitive to light touches. The capacitive sensors are more
sensitive, but they are vulnerable to environmental
interference. There are some sensing alternatives with
lighter hardware such as acoustic sensing and air
pressure sensing, but they can only detect with low spatial
resolution.
Given this existing gap in human-robot interaction, our
goal is to develop a low-tech low-cost touch sensing
alternative for social robots.
In this work, we introduce ShadowSense, where we pose a
camera inside a robot's body to capture the shadows created by
touching its skin. We think of shadows as a modality between
vision and touch - where it brings the high-resolution and
low-cost of vision sensing to the close-up sensory experience
of touch. It outperforms the traditional sensing arrays by
achieving higher resolution and gaining the ability to detect
both on-skin and over-the-skin activities.
The core interaction capabilities of ShadowSense
include detecting touch activity, classifying touch
gestures, and identifying the position of the touch event. A
range of gestures with varied scales can be sensed from
detecting a small fingertip touch to a human-sized full body
hug.
We implement ShadowSense with an inflatable robotic prototype. It
has translucent skin cover up the body and a fisheye Camera
inside capturing the shadows on both top and sides of the skin.
The robot prototype is designed and manufactured by the Cornell
Collective Embodied Intelligence Lab.
To detect and classify shallow images, we implemented densely
connected convolutional neural networks that take input of
shadow images and output a predicted label of touch
gestures, where you can see on the top left corner of the
video.
To track the touch positions we present a color
segmentation-based algorithm to track both touch positions and
the overall shadow positions.
We then quantitatively evaluated the performance of gesture
classification algorithm. We collected a total of 6000 shadow
images capturing six different interaction gestures, including
hug the robot, touch with the palm, point, punch, touch with
two hands, and not touch. We collected them under three
lighting conditions, including daylight, dusk and night. We
then split them to training and testing dataset, we train the
network with training data set and evaluated the accuracy on
the test set.
Here shows the results of the test accuracy and the confusion
matrix separately reported for each lighting condition.
It has the highest accuracy 96% under daylight, 92% under dusk,
and 87% under night. Overall it can distinguish between the six
gestures with relatively high accuracy, but it's negatively
affected with dimmer lighting condition.
This video shows a few potential Human-Robot Interaction
applications. With ShadowSense, you can easily turn an object
into a touch sensitive device with the simple addition of a
camera. Even just a balloon.
ShadowSense can be implemented on an inflatable guide robot
where the robot could respond to various touch gestures and make
movements accordingly.
Together with an internal projector, ShadowSense becomes
an interactive touch screen for inflatable soft robots.
Another use-case for ShadowSense is to provide a
privacy-preserving shield for camera based home interaction.
When robot entering a personal space, having a camera can pose
risks to the user's privacy. By physically covering the robots
eyes with a translucent material, a robot can still make
use of some interaction data in the form of user's shadows,
instead of their high fidelity images, like inferring what the
user is doing without collecting their actual appearance.
To summarize, ShadowSense is a vision-based touch sensing
method for social robots, and has many benefits compared to
the existing sensitive skin arrays. It can detect higher
spatial resolution with minimal hardware cost, it can operate on
different types of skins, no matter curved, soft or
deformable, and it expands the detection range to detect
gestures above the skin. For more information, please refer
to our paper.
That's the end of my presentation. I would like to
thank my advisor, Professor Guy Hoffman and my colleagues to
perform this work with me and the NSF for funding this
research. Thank you all for your attention, and I'll be happy to
take any questions