We introduce CapContact, a novel method to precisely infer the contact area between the
user's finger and the surface from a single capacitive image.
There are more than three billion smartphones in the world that use capacitive sensing for
touch input. Capacitive sensors measure the proximity of
close-by measure the proximity of close-by objects that change the electric field between
their sensor lines. The resolution of these sensors is determined
by their primary purpose which is to reliably detect touch events and to discriminate adjacent
touches. Therefore, a standard of around 4mm spacing
between grid lines has emerged in the industry. This pitch is insufficient, however, when
fingers move too close and the resulting intensities merge.
And it’s cases like these where we really care about precise contact areas and not just
touch locations. Of course, capacitive sensing was never designed to determine precise contact.
In the HCI community, we’ve been at this point before, where researchers played with
diffused illumination systems, but it wasn’t until the arrival of FTIR that we could work
with true contact areas and distinguish touches reliably.
And this is what inspired our work. With CapContact, we propose a single image
super-resolution method that estimates higher-resolution contact masks from lower-resolution capacitive
intensities. CapContact implements a generative adversarial
network that comprises a convolutional neural network with five residual blocks.
These blocks are connected to three consecutive sub-pixel convolutional layers that upsample
the image with a factor with a factor of 8. Due to the fully-convolutional architecture,
we can feed capacitive frames of any size into CapContact.
To train our supervised method, we recorded a training corpus with matched pairs of capacitive
touch frames and high-resolution images of the corresponding contact areas.
For this, our rig integrates FTIR into the capacitive sensor.
We first mounted a transparent touch panel upside-down onto the acrylic surface. The
connected digitizer resolved changes in mutual-capacitance at 16-bit resolution.
We added a sheet of acrylic on top, illuminated by two LED strips to complete the FTIR setup.
The webcam below the surface then captured the contact areas as they lit up due to touch
events. For the data collection, we recruited 10 participants
that performed 10 different touch gestures. In total, the full dataset has over 26,000
different pairs of synchronized frames. With the FTIR and capacitive images as input,
the currently existing default implementation with a threshold produces an area that is
way too large, just as expected. In contrast, CapContact reaches a contact
area size with less than 3% difference from the actual area.
Using our dataset, we also optimized a learned threshold that minimizes this error across
all samples, but it still produces a much worse outcome than CapContact.
In terms of discriminating adjacent contacts, the baseline often blends them, resulting
in an F1 score of only 0.69 whereas CapContact achieves an F1 score of 0.93.
Again, our learned threshold achieves an accuracy that is in between both.
As a side effect of reconstructing contact areas, the touch input locations we can derive
from CapContact are much closer to the true contact centroid than the center of gravity
of the capacitive measurements. For full details, please see the paper.
Because we’re working with super-resolution, we also conducted experiments on images at
just half the input resolution, so twice the pitch.
And still, CapContact produced contact areas that separated touches with an F1 score that
was much higher than the baseline achieved on full resolution images.
To double-check CapContact’s capability to work on half-resolution sensors, we ran
our network on Project Zanzibar, a sensor with a larger gridline pitch.
Here, CapContact robustly detected contact areas even under noisy conditions, successfully
rejecting false inputs and solely preserving actual touch contacts.
As you can see, the threshold-based methods suffered from missed touches as well as noise
due to the flexible surface. In conclusion, we have introduced the first
investigation of deriving actual contact shapes from mutual-capacitance sensing.
Our data-driven method CapContact estimates the contact area from a single capacitive
image, reliably discriminating adjacent touches and bringing input locations closer to the
center of true contact even when applied to a sensor with just half-resolution.
We think that CapContact opens up many opportunities for future touch devices, for example to reliably
distinguish hover and touch, to operate on much lower-resolution sensors with higher
pitches, and to produce higher touch-input accuracy.
You can find all materials on siplab.org, where we have released the dataset and code.