Thank you everybody for being here today.
Today I'm going to talk about my CHI'21 paper super-resolution capacitive touch screens
and this work was conducted during my time at Carnegie Mellon University with
my collaborators Xiangyu and Chris.
For a long time, researchers already set out to understand
how we can extend the capabilities of touch screens.
Here, in the end 2000s researchers tried to extend tabletops by integrating tangible user interfaces
and back then they used infrared technology to track the tangible object;
the same technology that's been used at the time to track also the finger.
However, in contrast to these infrared sensors today's technology which are capacitive touch screens the resolution is way lower.
So, for instance, one example here is this AprilTag with a size of 24 by 24 millimeters
and you can understand this as the bottom side of a tangible user interface, which was just placed on top of the touchscreen
and as a result, we see here four different orientations and variations of the capacitive image that is produced by the tangible itself
and here it is pretty clear that information got lost between the tangible that is placed on the screen
and the then recorded data information from the tangible
and here we really set out to understand and explore approaches
to upscale the information and really retrieve the information that is hidden in their capacitive images
and one approach, here, that we pursued in this paper is using geometric super resolution
an approach that is already long applied in fields like astronomy and biology
and the idea here is taking multiple raw images and then stack them combine them
and upscale them in order to retrieve information from all of the images combined
to then have a super-resolved image, which you can see here for the planet on the right side while seeing multiple raw images on the left side.
So, a similar approach we used now for the capacitive data
and that for the pipeline, we have the physical marker that is placed on the touch screen and
by motion induced by a human, we get multiple differently looking capacitive images.
Those images can then be upscaled aligned and stacked – on a high level speaking –
and then they give you a super-resolved capacitive image
and if you look at the image you can see the information clearly in the super-resolved image
however, not in the capacitive raw image.
So, to evaluate our approach we set out a number of experiments
and one of the experiments I want to present to you today
is here using multiple AprilTags that we designed and printed built as physical tangibles in different sizes c
as physical tangibles in different sizes four six and eight millimeters as one of the pixels size.
and we can really see how our approach outperforms in accuracy – so in recognition
over the baseline for the very small sizes of tangibles.
For traditional objects, however, this brings augmentation with it so in a follow-up we looked
at the geometric shape of an object itself such as in this example the coin size to recognize coins
and this already brings me to my conclusion I presented to you today
how geometric super-resolution works in principle and how we apply this method
to capacitive sensor data and therefore enable tangible recognition on today's capacitive devices
I gave you a bit of the insight that we gained from overlook from our evaluation in which I
showed you how we really can especially improve when there are small details in tangible objects
and I also gave you an introduction into two potential use cases which we see that
where really super resolution can shine to enable new applications in the future.
Thank you very much for your attention