- Hi, my name is Sang Ho Yoon
and I'm a PHD student at Purdue University,
and today I'm going to talk about our work iSoft,
a customizable soft sensor with real-time
continuous contact sensing and stretching sensing.
This work has been done with my colleagues
Ke, Yunbo, Guiming, Luis, and Sub,
with my advisor Professor Karthik Ramani.
Professor Ramani could not attend this year's conference
due to other commitments.
So, stretchable soft sensors have been explored
as promising input that can add interactions
to both rigid and elastic surfaces.
Still, the multi-step process
and the expensive equipment requirements
keep people from quickly making and deploying
inexpensive and customizable soft sensors.
So here we present iSoft, which provides users
to fabricate soft sensors inexpensively,
customize interface easily, and deploy them instantly.
For the related works,
our recent approaches have shown the potential
for reducing the fabrication complexity
and improving the design freedom for soft matter sensors.
Our work also maintains the sphere
of low cost soft matter sensors
by introducing simple fabrication process,
along with the customization software toolkit.
Electrical impedance tomography has been explored
in the robotics field to implement artificial skin
and in the HCI field for gesture recognition
and touch sensing.
We also utilize the EIT sensing to implement
multi-motor sensing for flexible
as is the stretch of our form factors.
Carbon elastomer has been used widely
for sensing purposes with its excellent
pressure resistant properties.
However, the long settling time requirements
after deformation and non-linear characteristics,
hinder researchers from utilizing
this material for real-time applications.
So, in our work,
we achieve real-time continuous contact sensing
by overcoming these issues.
Advancement in electronics and sensors
have promoted adding interactivity to different objects.
Still, the customization has not been fully supported
for the soft-matter sensors due to the
complex fabrication and design requirements.
Here, we propose a workflow that can
customize soft-matter sensors.
So, in our work, we focus on three things.
First, we put an effort on making a low-cost sensor,
and we enable rich sensing capability
with the proposed sensor.
And lastly, we offer customization
to entitle users with the design freedom.
So, how do we make the low-cost sensors?
First, we reduce the fabrication steps,
we use less equipment,
and we utilize the single-volume materials.
So, here is an overview of our fabrication process.
So we implement a carbon filled silicone
from the wicker, which works as a
pressure resistive sensor.
First we mix the two parts components,
and we apply this to the heat stabilized
filament using a pallet knife.
And we can either use t-shirt heat press
or a toaster oven, with a steel jig.
To be used in an oven we had to use the jig,
in order to keep enough pressure during the curing.
And the material can be cured in 140 seconds
using t-shirt heat press, which is quite fast.
But for the oven, it takes about 60 minutes
to get fully cured.
Our sensor is then fabricated
with no additional material processing,
and it is ready to be shaped into arbitrary 2D shapes,
and we can also reproduce the filum with the textile,
to embed the sensor onto the textile substrate.
So here's our overview of implementation
of our electronic parts for EIT sensing.
So we mainly utilize the 16 channel multiplexer
with a micro controller.
We attach the crown source and the mini amplifier
to provide fixed current and read the board measurements.
Here we apply less than point five milliampere
for the fixed current and observe frame rate
greater than 50 hertz,
and the cost of making these electrical components,
including microcontroller was about $30
and this can be further reduced over the
commercial production phases.
So with the local sensor what kind of sensing
capability we are providing?
So we provide discrete contact localization
which is discrete touch, and we provide
continuous contact localization which is a continuous touch
and stretching sensing.
So we put extra effort on providing real time sensing
where our sensing introducing here can be used
in real time with real applications.
So for our contact localization we employ
electrical impedance tomography,
so basic workflow is find an element model
on the given shape, processing sensor values
and reconstructing images based on these measures,
and applying holograph detection to
locate the fingertip contact.
EIT technically estimates the contact position
based on the resistance distribution from the conductive
material, so here an inverse program analysis is used
along with measurement from the sensor boundary.
In our work we focus more on the processing sensor values.
So previously the difficulty of providing real time
sensory with a pressure resistant material
is due to the rebound elasticity, so when the contact
pressure was applied and released to the carbon filled
silicone, it causes a long settling time and
small shifts in baseline values.
This makes it hard to do the real time
continuous contact localization.
To overcome the baseline shift issue, one way you could
would be using fast EIT, where the system obtains
the baseline calibration every frame.
However as you can see from this video,
it lasts for a short amount of time but it doesn't
last for the case like when the fingertip,
when we hold onto the same location, the fingertip
doesn't create a new deformation after a short moment.
Therefore it is crucial for the system to aware
the fingertip content.
So traditionally in EIT, the neighboring method
has been utilized for measuring resistance of the material.
So here this current is faster to electrodes
and what is differential is measured successfully
throughout these electrode pairs.
To enhance the sensor with the fingertip contact awareness,
we added a capacitive sensing channel to check
the presence of the fingertip content, and this
allows system to update the baseline only during
the presence of the fingertip content.
However still there is another issue to fully provide
continuous contact sensing.
As you can see from this video
even with the awareness on the fingertip content,
the register deformation makes it hard to get the
dynamic content localization, as you can see there is
some register deformation during the dynamic
fingertip movement, therefore it is also crucial
to detect the movement of the fingertip.
So you observed that the average of the overt
channel measurements, of the change in output voltages,
are increased subsequently upon fingertip movement.
So we utilize this change in output voltage as in indicator
to detect the fingertip movement,
and we use the loading average of five frame
which is point one seconds in real time,
are used to provide the stable and lowest interaction.
So by employing both contact and movement
detection we provide continuous contact
localization in real time.
And as you can see from this video any form of
continuous contact localization can become feasible,
and more details on system workflow
can be found in the paper.
So we also support stretching sensing where we use
a regression model to recognize
different levels of stretching.
Currently we only support stretching at loading instance
only due to the hysteresis effect, where the sensor
values are different at loading and unloading instances.
So with up to 40% unidirectional stretch we can
form a robust regression model.
We also test our sensors for the stability issue
and we tested the sensor up to 1000 stretches
and we found that there is a sharp increase
in the resistance of the first 50 stretches
after initial fabrication.
As found in previous research, this is due to the
breakdown of the internal carbon plate network
after the initial calibration of this
carbon plated silicone.
So how do we support customization?
First, we provide a customization software toolkit
for users to design the interface.
We also provide a hardware approach where the hard design,
we support various shapes and surfaces.
So we provide proof of concept toolkit
where it generates a guidance image
which users can look for in designing the sensor
and deploying electrodes.
Users can deploy their own personal interface
using this toolkit.
So here is a short video of where users customize
their interface, so first the users load the image,
and they can design their own interface by
drap and drop the different UIs,
so this is an example of the music controller.
Then the toolkit generates the guidance image
for pressing electrodes, and users can use this
guidance image to shape the sensor and
as they're placing the electrodes.
And after placing the electrodes the user
can now this personalized functional soft sensor instantly.
More detailed implementation on this and
algorithms for toolkit can be found in the paper.
So here we summarize the technical evaluation.
So we tested the contact localization accuracy
in different sensor sizes and number of electrodes.
Here the sensor segments was point eight millimeters.
First we tested the activation force requirement.
Since we are based on the resistive reading,
this requires some force to activate the sensors,
and we tested it on all sensor regions and the
range of the contact force requirements we found out
to be around one to three Newton forces.
We found out that this is the comfortable fingertip
pressure range according to the previous paper.
And for targeting error, it came out to be around
4 to 7.5 millimeters for 16 electrodes,
and 10-18 millimeters for 8 electrodes.
One thing we noticed is that there is an improvement
of 10 to 30% in errors if you're only focusing
on 80% of the sensor area.
So we also checked the feasibility of multimodal sensing,
of the contact sensing and stretching sensing.
First we looked at whether contact pressure
can activate the stretching sensing.
As shown in this figure, the sensor values
caused by contact sensing is much less than that
of stretching sensing.
Therefore it is very hard to mistrigger the stretching
sensor due to the contact sensing.
We also investigated the contact localization
under stretching, so we
we used a test sheet to apply uni directional stretch
up to 40%, and we found that there's a slight increase
in localization error, which is about 1.1 millimeter
in difference, when the material surface stretches for 40%.
So this shows that the localization still works
robustly under high strain.
So now we evaluated our sensor with the real users.
So first we looked at the discrete contact performance.
So we used two by two centimeter grid on different sensor
sizes and we looked at both the targeting accuracy
and the distance errors.
So the overall discrete targeting accuracy
came out as 96%.
As you can see it's these blue bars,
and the red bar indicates the distance error,
which came out to be 8.5 millimeters.
So the high targeting accuracy across all sensor sizes
indicate that our prototype strength
was discrete contact localization,
and we also looked at, we also explored the system
accuracy for continuous contact localization.
So here we used vertical and horizontal strokes
where users do the swiping vertically and horizontally
on our sensor regions.
And the average distance error about 8.7 millimeters,
for the ending contact points.
Here the ending contact point meaning that we
only look at the distance error at the end points.
If we look at the overall stroke lengths we found
that is a little bit higher because there's an error
between the initial contact point and the ending
contact point, and this came out to be 13 millimeters.
The only other thing we noticed is that
the stroke distance errors were less than 10% of the
sensors size for different sensor sizes.
So we also evaluated the participants targeting
controllability with the stretching sensing.
So our goal was to conform how many stretching
resolutions users can handle.
So we evaluated with a stretching range between,
so we used the stretching resolution from two to ten levels
and we found that the average targeting accuracy
was greater than 90% if three or fewer levels
were provided, however the performance drops
sharply as four or more stretching levels were provided.
So, therefore, it is recommended to limit the interaction
to three or fewer levels for
the best stretching interactions.
So what can you do with our prototype?
So I'm going to showcase five example applications
with our prototype.
In this example, a user can design and make
various shapes of interface using iSoft.
As you can see there are arbitrary 2D shapes of sensors,
and these sensors can work instantly with
discrete and continuous contact sensing capabilities.
Also in this example we attached our sensor
the adjustable lamp arm.
By attaching our prototype, the lamp arm becomes
capable of sensing bending and discrete contact,
and swiping, and we can also squeeze our sensor
to turn the lamp off, and this example demonstrates
the versatile sensing capability.
In this example we also utilized our prototype
to design a sensor that can wrap around a 3D object.
So users can load a 3D model and design the interface
like they're doing in the 2D, and here it generates
a guidance image from the toolkit.
The fabricated sensor can be attached onto the cup
with adhesive, and the tumbler instantly turns into
the mission controller, and here as you can see
users can control different areas to activate the functions.
So we can also fabricate iSoft using textiles
as base substrate, and in this way we can bring
interactivity to the clothing.
And various parts of clothing can become interactive
such as the arm sleeves and the pockets.
Our prototype can be also attached or embedded
in a volumetric fabric such as neck pillow.
Here users are squeezing on the different locations
to change the channels or squeezing and stretching
on different sides to change the volumes,
and this demonstrates the use of multimodal sensing
utilizing users' physical motions.
So I'll conclude my talk with the limitations
and future work.
So in our current setting our physical
sensor size is limited.
We tested up to the letter size, however the sensor
can be fabricated using the industry heat compression
which is much larger than what we are using.
Also applying soft sensor conformably onto
complex 3D surfaces remains unresolved.
So the potential solution would be providing
efficient or flattening complex 3D surfaces into 2D pieces.
And current prototype lacks sensing
other physical modalities such as bending
and multi directional stretching, and also
we are not currently supporting
concurrent multimodal sensing.
So in our current work we are in the process
of exploring on supporting other physical modalities
as well as concurrent multimodal sensing by
formulating a different version of our system.
With this future approach we expect to provide
richer and more robust customizable soft sensors.
Thank you for listening and
I'd like to answer any questions.
(clapping)
- Great talk, I really like the lamp example at the end.
It's a clever use of stretch in addition to touch.
My question relates to touch while stretching.
So you tested it on uniform stretching, sort of in the lab,
does it work as well if just a person is stretching
in sort of a non uniform direction?
- Yeah so currently, in current work we only focus
on the unidirectional stretching where the direction
of the stretching is fixed, but we are in the process
of applying the motion learning to actually
recognizing the multi directional stretchings.
- Okay and does it work as well in the touch
when there's no hard surface under?
Sort of it you're touching fabric it's kinda gonna
go through, it's gonna stretch as you touch.
- So the basic sensing principle of this material
is if there is a strain on the material
it gives a change in the sensor signals,
so even if it's in the air or
(timer ringing)
hold onto two different points and do the...
- Sorry.
- Yeah.