- So thank you for joining this talk.
I am Patrick, and this is Adwait.
And today we present SmartSleeve.
This is a joint work between the Media Interaction Lab,
the Saarland University, and Google.
The main motivation of this work
is to use the physical prototypes properties of textiles.
Clothes or textiles are most often soft and comfortable.
They are stretchable and non rigid.
And we wear them on a daily basis.
Therefore it seems to be the ideal medium to interact with.
By researching the related work we discovered
that the author either concentrate on surface gestures
like taping or swiping, or on specific deformation gestures.
Two fully functional prototypes have been shown
with Project Jacquard and GestureSleeve.
However the author concentrate on surface gestures only,
and do not fully take advantage
of a flexible textile sensor.
Other projects like Pinstripe and Grabbing at an angle
demonstrate how deformation gestures on textiles
can be detected and used.
However most of the systems are designed to explore
one specific gesture type only.
SmartSleeve, in contrast, enables the detection
of surface gestures as well as deformation gestures
with the same sensor.
Before we dive deeper let's see how SmartSleeve
performs in action.
In this video you can see how user interacts with
the sleeve, and the corresponding force image
in the back.
On the right side of the screen the detector gesture
is displayed.
Let's start with the implementation of the sleeve.
Beginning with the textile sensor material.
The connection between the soft textile
and the rigid driver electronic and our improvements
regarding cross talk.
SmartSleeve is a pressure sensitive textile input sensor,
which is based on flextiles sewn in a sleeve form factor.
The sleeve consists of three layers of fabric,
all equally by directional stretchable.
The top and the bottom layers are the connection layers,
which consists of alternating stripes
of conductive and non conductive fabric.
The two layers are placed (mumbles) over each other
to form a grid.
The middle layer consists of a pressure sensitive fabric.
Applying mechanical stress onto the layer causes
a resistance drop, which can be measured.
Let's have some numbers.
The sensor can detect forces between 50 and 500 gram,
with a spatial resolution of one dot six sensors
per square inch.
In total the sleeve consists of 360 sensors,
and weighs about 124 grams,
including the connection wiring.
Due to the reason that SmartSleeve is a non-rigid sensor,
we have been looking for a connection between the soft
textile and the rigid measurement electronic,
which doesn't limit the flexibility of such a sensor.
And what we came up with was an unobtrusive
sewn-based connection using small diameter wire,
which can be easily sewn into the textile.
As wire we used (mumbles),
which has a small diameter, isolated wire,
usually used for repairing
and corrected printed circuit boards.
The isolated enamel coating can be easily removed
with a soldering iron at a temperature
between 400 and 430 degrees.
The small diameter of only 0.15 millimeters
makes the wire very deformable and therefore ideal
for SmartSleeve.
The wire is really robust.
It even withstands sewing with a normal off the shelf
sewing machine.
In order not to limit the flexibility of the sleeve,
we have evaluated three different
stretchable stitching types,
which are super stretch, double overlock, and zig zag.
All of them work probably, however one performed
better than the others.
Super stretch raised more tension to the top yarn,
and therefore it can tear more easily.
Double overlock applies less tension to the yarn,
but unfortunately this stitch uses more wire,
which makes the fabric stiffer.
Therefore we recommend zig zag stitch
because it doesn't apply a lot of tension,
nor use a lot of wire.
Its simple pattern and stretchability,
maintenance the comfort of use.
As mentioned before, SmartSleeve is based on a resistive
textile sensor which suffers from crosstalk.
This failure affects the accuracy, the measurement
at the recognition system.
On the right figure you can see an equivalent circuit,
an equivalent electrical circuit of a three by three matrix.
Every resistor can be seen as a sensor.
Crosstalk is a phenomenon by which one sensor
in the circuit creates an undesired effect
on another sensor.
This crosstalk phenomenon is caused in a way
such a sensor metrics is formed.
In order to eliminate those failures
we explored and evaluated different read out methods.
Therefore we used a three by three matrix
where we pressed six sensors equally.
This force pattern allows us to evaluate
the distribution of crosstalk along the columns
and the rows of the matrix.
All read out methods have been tested
with the same sensor matrix.
First we analyzed how our system behaves without
crosstalk reduction.
And we had an average error of 34.5%.
Next we evaluated electronic solution to reduce crosstalk.
These solutions used electrical properties
of an operation amplifier.
These methods are also called zero potential method,
grounding and vertical grounding.
In both cases that we evaluated,
we achieved an average error of about 11%.
Further, we evaluated mathematical solutions
to reduce crosstalk.
Which was proposed by shoe and others.
In this approach we add fix constant resistors
for every single column.
This allows us to measure the effects of crosstalk
of the under determined matrix.
In comparison to the other methods,
it does not suppress crosstalk.
Instead calculate the crosstalk distribution
for every single sensor.
With this method we achieved an average error
of less than one percent.
Therefore we used that method for SmartSleeve.
The sensing properties of SmartSleeve
enables a wide variety of interaction techniques.
Based on prior related work and an iterative design process
we composed a list of 22 different gestures.
We further categorized them into surface
and deformation gestures.
The surface gesture set consists of 14 2D gestures,
which are performed on the textile.
Similar to conventional touch screen interactions,
we provide single finger, as well as multiple finger input.
The deformation gestures consist
of eight different gestures.
These gestures are based on deforming the textiles
in more dimensions.
Taking the advantage of the sensor's flexibility.
We explored what kind of associations people have
with these interactions.
For examples twisting a force rotational control,
where the analogy is to a physical knob.
Bending allows an input modality even when the hands
are occupied.
And twirling the fabric around the finger
requires intentional coordination.
It uses the metaphor of a reminder knot.
We broke a wide variety of interactions on (mumbles).
Enabling an expressive connection between the physical
and the digital world.
Now Adwait will present how these gestures
can be detected.
- Thanks Patrick, the common problem with machine learning
techniques is that it requires a lot of training
for every gesture that needs to be detected.
Our aim here is to mitigate this tiring process.
Between the classifier on a frame by frame basis,
in the next step we convert the raw sensor data
into a gray scale force image.
Further we separate the foreground and the background
of each pixel to find the region of interest.
We calculate the pressure information
using the raw sensor data that helps us to detect
the magnitude at any given instance.
Furthermore we do bilinear up-scaling and caution filtering
for blob detection.
As you can see in these images,
the nine trained gestures are highly discriminative.
Therefore we only calculate the histogram and the
properties of central control bounding (mumbles),
including height, width, area, and parameter
as features for the classification.
Further, we use the extracted features as an input
for the SVM classifier.
This gesture classification is position invariant.
Therefore training each gesture at different location
is not required.
Also due to the contour detection mechanism,
we can detect the position on sleeve where the gesture
is performed.
So far we have trained nine gestures,
and we use a set of heuristics to extend our learning
base algorithm to detect untrained gestures.
In total we are able to derive 13 additional gestures
from only three trained ones.
For this, we use simple yet robust fundamentals
as heuristics.
For example rub gesture is a combination
of detecting a hand that we trained
during the classifier and the oscillating pattern
of the blob central coordinate.
Therefore we use the concept of first order derivative
to identify this motion.
Similarly, for shake we use the classifier's prediction
of grasp.
Spread is derived from the finger classification.
There is a linear change of bounding box area
for spread, the area of the bounding box increases.
While for the close, the area of the bounding box decreases.
Furthermore, for detecting the direction,
up, down, left and right of the gesture,
we save the consecutive blob central coordinate
in a buffer and compute the slope.
To evaluate our hybrid gesture detection algorithm,
we conducted two empirical studies.
The participants in both experiments have been from
a local university.
SmartSleeve is a custom tailored device,
therefore we recruited only three males and three females,
with almost identical arm dimensions,
with ages from 23 to 25.
All participants were right handed,
and used 2D interfaces like smartphones and tablets
on a daily basis.
None of them had experience with smart clothing before.
With the first experiment, we wanted to evaluate
the accuracy of our learning based algorithm.
Furthermore, we wanted to investigate
if the gestures can be detected throughout the sleeve.
Therefore, we evaluated the nine trained gestures.
The participants were instructed to perform four trials
of each gesture of a marked position
to train the classifier.
Next the participants were asked to perform four trials
of each gesture type on the same location,
and four trials of each gesture type on arbitrary location.
To avoid training effects, the gestures were shown
in a randomized order.
The results of the first experiment have shown promising
accuracy of 92% when training and testing was performed
on the same location.
For different testing location, the accuracy was 87%,
which indicates it is position invariant.
These results are only from four trained trials,
and we believe with more training,
the accuracy will improve.
Our next experiment addressed the accuracy
of the heuristics.
In this study we evaluated the 13 derived gestures.
The same participants from study one performed each gestures
five times in a randomized order.
Overall, we had an accuracy rate of 84%.
Spread achieved the highest accuracy of 97%.
Unlike Project Jacquard which has rigid cuffs
in a jacket form factor,
our design choice for SmartSleeve is to have a soft
and comfortable material,
which can be directly worn on the skin.
Thus it needs to be flexible.
As a result of this design choice,
swipe left with full hand was 57%,
as the textile deforms while performing the gesture.
Now we would like to present some of the application
scenarios which demonstrate a combination
of the previously discussed interaction techniques.
Here the user interacts with a video player
using bend gesture to trigger play.
A finger swipe down gesture to fast forward the video.
And a figure swipe up gesture to rewind the video.
Whereas a hand swipe gesture is used to switch
between videos in a playlist.
The rotational control analogy provided by twisting
can be used to control the volume of the audio.
To mute the audio, or as a state change,
the user pushes the sleeve up.
In the next application, the user performs conventional
touch gestures as input (mumbles).
For example, spread to zoom out, and pinch to zoom in.
Now we would like to discuss the future work
that we are currently working on.
While the three layer approach is a practical solution
for resistive sensing, it was problematic
in solutions when the user grabs only the two top layers
while performing the gesture.
Therefore in the future work we would like to reduce
the layer stack to a single layer.
Additionally, we will like to integrate the front spatial
sensing resolution to fit users need.
Also a designing tool to create
user customized interactive garments.
Furthermore, we would like to give visual feedback
to enable input as well as output on the textile itself.
Summarizing, with the SmartSleeve we have shown a soft,
and flexible textile sensor
that allows free form manipulations.
The continuous pressure sensing provides
expressing input capability.
It enables the detection of 14 surfaces
and eight deformation gestures.
Overall this brings us one step closer
to our vision of using clothing as an always available
input medium to the digital world.
Thank you so much for listening,
we'll be pleased to answer any questions you may have.
Thank you.
(audience applause)
- [Yong] Hello, Yong Jeong from (mumbles).
Great work, congratulations.
So just one question.
Since you are detecting a finger on top of the sleeve,
do you also get the signal from the skin underneath?
For example your arm.
And how do you like make your system robust
so that it doesn't account of the signal
from the arm?
- Just to get that right.
So you mean if we are detecting a finger on the arm,
and there is any other signal that is happening
at the same time?
- [Yong] Because the sleeve is sitting on top
on the arm right?
And there is like your arm underneath the sleeve.
Will it change the signal?
- No, so when you press the sleeve it only appears
the single blob.
We do perform a threshold that removes
the noises that happens.
- [Yong] Okay, thank you.
- We have time for another question.
Any other questions?
I actually have a question for you.
I've been wanting to ask.
So your examples, I don't know if everyone can hear me.
Your examples were mostly about explicit interaction
on the sleeve with gestures and so on.
But with fabric we could have a whole body suit
covering all of our body.
So do you think at all about how you might do activity
monitoring or other kind of passive sensing
that's not using it as a touch surface per se,
but using it to monitor the body in motion,
and could be useful for athletic training
or physical therapy or other things like that.
- Yes prior work have shown that this is doable
with the textile sensor.
For example for detecting breathing, as well as like
bending the prosthesis.
And well that's an interesting topic for the future as well,
for sure.