Hello Everyone, My name is TeYen Wu.
I am a third year phd student From XDiscovery lab
at Dartmouth College.
Today, I am going to talk about Capacitivo, Contact-Based Object Recognition on Interactive
Fabrics
using Capacitive Sensing
This work was done in collaboration with two undergraduate research interns, Lu and Yuji
as well as my advisors teddy and xing-dong
Before going into the detail of this work, I want to briefly show you what Capacitivo
is
and what it can be used for
Capacitivo is a capacitive sensing technique that makes fabric aware of the objects it’s
in contact with.
This includes things like food items, and dinnerware.
It can also detect the change of what is inside objects, like containers or cups.
For example, it can differentiate between milk, coke, apple cider, cold and hot water
in a regular glass cup
And as many everyday objects around us are made or covered by fabric, there are a numbers
of applications that are enabled by our sensing technique
Such as turning a coaster into a diet tracker with our smart textile.
or a smart tablecloth for recipe helper
Next step, add a slice of cheese
Although there are a number of techniques to enable interactive fabrics,
the one most related to Capacativo, is Tessuitivo by Jun Gong et al.
Tessutivo demonstrates an interactive fabric to recognize metallic objects based on inductance.
In contrast, our work focuses on nonmetallic objects that are also very common in daily
life.
Our work complements existing research with a new sensing technique that is based on capacitance.
Next, I will explain how the sensing works for Capacitivo
When a object is placed on the textile sensor that is made of conductive electrodes,
the capacitance among the electrodes will change due to the permittivity of the object.
And different materials of the object will bring up different permittivity, resulting
in different capacitance.
Thus, Capacitivo recognize objects by measuring different changes in capacitance.
and next, let’s dive into more detail of the sensor design
Our sensor is composed of diamond-shape electrodes,
that are connected by rows and columns
and there’s a ground plane on the bottom, that is similar to the common design of touch
panel,
With this setup,
there are 2 types of capacitance occurring on the electrodes,
They are mutual capacitance
and self capacitance.
Mutual capacitance is the capacitance between the adjacent electrodes.
Self-capacitance is the capacitance between the electrodes and the ground plane.
Both are affected by a contacted object.
To allow the signal a more pronounced in response to the small change in the capacitance,
we used a resonance-based capacitive sensor to measure the aggregation of them.
However, this approach is problematic.
The sensor readings at a location interfere with all electrodes connected in the same
row or column due to self capacitance.
To solve this problem, we also measure the mutual capacitance to capture the contact
area of the object,
and then gather sensor readings only in the contact area
Next, we need to determine the proper size of the electrodes
and the separation distance between electrodes, which are two factors that affect sensor sensitivity.
Thus we conducted two experiments
In the first experiment, we tested electrodes with sizes of 7mm, 14mm and 21mm, with a fixed
separation of 2mm.
The results are revealed in the left side of table.
Clearly, 7mm is the best option among three sizes considering the balance between the
sensor sensitivity and resolution,
Next, we conducted a second experiment, where we tested the separation of 2mm, 3mm and 4mm,
and a fixed size of 7mm.
The results are shown on the right side of table.
Surprisingly, the separation of 4mm is the best option since it increases sensor sensitivity
significantly but without largely impacting 2D resolution
Thus, through these two experiments, we determined the size of 7mm and the separation of 4mm
for our implementation.
Next, I want to talk about our DIY approach for fabrication
First, we used an iron-on adhesive to bond a conductive fabric to a textile substrate.
Then, we used a desktop cutting machine to cut out the electrodes and row connections.
After cutting, we reheat it again and then peel the unnecessary pieces off.
so the pattern is clear.
Next, we need to connect the column electrodes
To connect the column electrodes, we use a sewing machine or hand stitching to connect
the conductive thread for each column.
Then, we attached a conductive fabric to the bottom as a ground plane
the sensor is completed
Next, I am going to describe how we accomplished object recognition using the textile sensor.
When an object is placed on the sensor, the sensor reports a 12 by 12 capacitance value.
Based on our observation and initial tests, we extracted 86 features, which can be divided
into two categories (click),
shape related features like the number of covered pixels
and material related features like mean and median.
After the feature vector is packed, we then feed it into a random forest model since it
has been found to be accurate, robust, scalable, and efficient in many applications.
We used the output from the model as our prediction results.
In the following demo video, I am going to demonstrate our sensor working in real-time.
The detected item name will be shown on the left side of screen.
The heatmap image (click) of the capacitance footprint will be shown on the right.
We can detect different types of objects, like those in an office, such as hand sanitizer, AirPods, lipstick, hard drive, credit card, and another credit card.
We can also detect the objects in a living room, such as candle, book, unwatered plant, and water plant.
We can also detect the glass and if water in the glass.
We can also detect detect objects in the kitchens, such as bowl or bowl of soup, cheese, and salt
We can also detect multiple items and even different types of liquids.
To validate the sensor’s accuracy and robustness, we ran a ten-participant user study.
We tested twenty everyday objects, including kiwi, avocados, grapefruit, credit card and
so on
Each object was repeated for ten times in a randomized order.
We pre-trained a model three days before the study and we ran the real time recognition
in the study
In turn, our system can achieve an overall accuracy of 94.5%
This is an encouraging result, especially for the items which have a similar shape but
have different content or materials, such as kiwi and avocado
In addition, we also evaluated how well the system can recognize different types of liquids.
For this evaluation, we recruited a volunteer
and asked her to test 6 types of liquids,
including cold water, hot water, coke, apple cider, milk and beer
Each drink had 20 samples for training and 20 samples for testing
The result showed the overall accuracy of 90%.
The worst case, which was beer, if removed, results in a higher accuracy, of up to 96%
Aside from these tests,
we also tested the performance of the sensor in a vertical placement, similar to a pocket.
The process is the same as the one described previously.
We had 1 volunteer and 20 samples for training and testing.
We included 8 objects can be put into the sensor, such as AirPods, Lipstick, Kiwis,
Avocados, Cheese, a Discover Credit Card, a JCPenny Rewards Card, a Book, and a External
Hard Drive
The overall accuracy is only 70%, which is likely because our sensing technique is contact
based.
When the sensor is placed vertically, the contact between the object and the sensor
cannot be guaranteed since the object may partially lean against the sensor
or due to the deformation of the sensor itself.
If we remove the least 3 performing items, the accuracy can raise to 87%
In addition to the applications shown in the beginning, I want to show you some more demo
applications of our sensor.
For example, if we integrate with an IoT speaker, our system can remind a user to take a lipstick
left on the table
When a user registers or shops at a website for the very first time, their tablecloth
recognizes the credit card placed on it and automatically fills in their card information
Further, the tablecloth can also detect if the user’s table plant is in need for watering,
so the user can be reminded when the soil is dry.
Finally, the system can remind the user to clean the table after the empty bowl used
for soup was left on the table for several hours
With that, I would like to conclude our work
with three take-home messages.
We present a non-metallic object recognition technique on interactive fabrics using capacitive
sensing
We measure the accuracy of our technique with several experiments
We demonstrate unique applications enabled by sensing technique
thank you for taking the time to listen.