Our paper TapNet is about the design, training, implementation, and application of a multi-task
learning CNN for off-screen mobile input. My name is Michael and this is a collaboration work
with Yang, Nazneen, Alex, and Shumin. TapNet is a multi-task network that learns from cross-device
data. It has multiple input branches, each of which accepts data from one specific device.
The multi-branch architecture enables joint learning using cross-device data.
TapNet is also a multi-task network and each output branch recognizes one tap property,
including tap direction, finger parts, and location. This multi-output
architecture allows for the understanding of the interrelationship among multiple tap properties.
And here is the demonstration of that about the TapNet results. The heatmap indicates the tap
location. The color denotes the tap direction. Green for back, red for front, and blue for side
taps. The texture here shows the finger parts and the dotted pattern indicates the tapping
using fingernail and then non-dotted pattern shows the tapping with finger pad.
And here is a glance at the accuracy and sensitivity.
TapNet achieves roughly one-tenth of the screen diagonal for tap location estimation.
Gentle taps can be recognized. We also include diverse non-tap motions in training. As such,
TapNet can distinguish tap events from motions such as tap release, phone shaking, etc.
This is the TapNet pipeline. Its input includes IMU signals and device vector. The device vector
here indicates the IMU install location as well as the phone size. The core of the TapNet is a
multi-layer convolutional neural network. It has multiple convolutional layers, max pooling
and for each output branch it has multiple fully connected layers and each branch is specific for
one recognition task, including tap event detection, tap direction classification,
finger part classification, and tap location classification as well as location regression.
And we take turns to train these two parts of the TapNet. For the tap property recognition branches,
we use the diverse tapping action with annotations about the direction location and tap finger parts
for training. And for the tap event detection branch we use the tap data as well as the non-tap
data for training. We collect a one-person training data set, meaning that the TapNet is
trained on the data from only a single person. And then we collected a multi-person test set.
Our model is evaluated on multiple persons. We use the weighted average f1 score to measure the
classification tasks. We use mean absolute error and r square to measure the accuracy of their
regression tasks. TapNet can either perform on par with or outperform the prior arts.
A close examination of the normalized confusion matrix of the 35 class tap location classification
shows that the neighboring region above or below the target has an index of offset of 5,
and the three parallel lines here along the diagonal indicate the TapNet either predicts
correctly or to nearby regions. There are a few takeaway messages from this paper. For example,
the training strategy using one-person data can be effective for gesture recognition systems.
Multi-task learning can be more data-efficient in our tasks, in particular, when there
is only limited amount of training data. And cross-device training with different form factors
increases the efficiency of data utilization. We also see that the one-channel CNN can
achieve cross-channel signal alignment more efficiently than its multi-channel counterpart.
And also many off-screen applications can be promising
and full of potential. Please refer to our paper for more details. Thank you.