hi everyone thank you for coming to my
talk today I'll be presenting saw sense
I first want to quickly contextualize
this work within a vision for
intelligent surfaces
there has been a lot of work looking at
how to enable gesture interfaces on
surfaces such as recognizing swipes
happening on the surface of a table
and additionally towards ubiquitous
sensing like understanding what
activities are happening in the home
and most recently there's been work
towards computational materials such as
having the actual fabric of a category
part of the fabric of computation
in all these cases we want to look and
feel the objects to remain the same so
how do we get there
at a physical level there are generally
three types of waves that are launched
when a finger or an object hits the
surface there are surface acoustic waves
that travel along two dimensions on the
surface of the material their
traditional bulk vibrations that travel
through the entirety of the material and
there are sound waves that travel
through the air
however there have been some challenges
with detecting surface events with
previously used sensors each sensor type
has had to consider some trade-offs for
example microphones which are easy to
deploy and very sensitive have had to
deal with also capturing environmental
noise
accelerometers are robust that noise and
easy to deploy end up having limited
bandwidth and range
and geophones while sensitive and also
robust the noise are bulky and must be
vertically oriented
this work explores the possibilities of
using vpus which sends surface waves
that embody all three of these
characteristics
bpus or voice pickup units and they're
originally designed as bone conduction
mics for in-ear headphones they were
designed to pick up waves that travel
through the body while rejecting in-ear
sounds from the environment bpus add a
sealed Mass spring to a traditional mems
microphone which transduced surface
acoustic waves to pressure waves that
can be picked up by that microphone
this allows you to have the bandwidth
and sensitivity of microphones while
only capturing surface acoustic waves
and not in-air sounds
we repurpose these sensors to capture
saws on a variety of surfaces
in this paper we performed a robust
comparison of four different sensors but
here we're going to demonstrate one to
provide an intuition to the advantages
of epus
your fingernail makes a single tap on a
wooden workbench while news is playing
on a speaker in the environment only a
vpu and a microphone can produce a high
resolution capture of the nail tap yet
the microphone is also capturing
background noise the news playing
highlighting yellow
for a more direct example we will play
two clips where a person Taps and swipes
on a desk 1.8 meters away from the
sensor while music is playing in the
room here the first clip is a
traditional microphone
and here's saw sense in the same scene
in short sonsense provides a unique
sensor that is robust to noise yet is
very expressive even when the event is
at a distance from the sensor
we'll now talk about how we evaluate saw
sense across two domains of HCI gesture
recognition and activity recognition
we first evaluate saw sense as a gesture
input system
we selected three subtle gestures finger
tap swipe and fling
and also evaluated their fingernail
equivalence which are more in line with
prior works
the figures on the right show each
gesture and the frequency signature
showing how each gesture's signature is
visually visibly separable
We performed a 10 user user study on a
wooden table including persons with long
nails and acrylic nail extensions
we found promising results we first
performed a teasing analysis which is
one way to represent how separable each
class is from this plot we see very good
clusters with slight overlap from the
more subtle finger fling and nail fling
gestures
our confusion Matrix corroborates these
results having the most confusion
between nail flank and finger fling
overall we found a greater than 97
percent per user accuracy and an across
user accuracy with augmentation the
details of which can be seen in the
paper
in this clip we'll demonstrate saw
senses real-time predictions with the
laptop actively classifying the gestures
starting with finger tap
finger swipe
finger fling
nail tap
nail swipe
and nail flank
we also evaluate saw senses ability to
recognize activities in the kitchen
we selected 16 various appliances
objects and actions Each of which have a
visually distinct frequency signature
they were evaluated both on a wooden
counter and a metal kitchen island as a
10 round leave one round out evaluation
we evaluate saw sense in a similar way
to our gesture evaluation
first looking at our teasing plot the
categories can be seen to be well
separated which matches our intuition
given how different for example an air
fryer is to a water boiler
we found over 99 accuracy for both wood
and metal surfaces and performed a
preliminary evaluation generating
synthetic features from one material for
another material and found 98 accuracy
additionally since saw sense is robust
to environmental noise we can perform
real-time classification of kitchen
events even when music starts to play in
the background
hey Google play classical
got it here's a Spotify playlist called
classical Essentials
foreign
so in the previous two evaluations we
looked at more traditional sensing
modalities on Rigid 2D surfaces like
tables but we knew how expressive epus
are and we wanted to explore where else
saw sense could be used by placing only
one VPO on each of these objects we can
enable multiple kinds of interactions
like making the entire curved surface of
a coffee machine touch sensitive
or supporting playful interactions on a
toy with a textured 3D printed bone
or by placing them on speakers that are
actively playing music
or on soft surfaces like a fabric couch
to make the arm touch sensitive
or on the sleeve of a jacket to enable
wearable applications
given the length of this talk I wanted
to highlight some of the things in this
paper that we couldn't get to
we did an information power analysis to
craft custom features for sauce
we detail our augmentation approaches to
boost across user accuracy
we looked at how to generalize saw sense
across materials so metal to wood or
wood to metal
and we performed our evaluations with
multiple machine learning approaches
showing that saw senses machine learning
and variant
and finally we looked at a lightweight
saw sense that in the future could be
deployed on mic processors
in conclusion surface acoustic waves
present a novel sensing approach
they use as a small and easy to deploy
vpu
where the wide bandwidth enables Rich
activity recognition in addition to
gesture sensing
and when paired with a lightweight
machine learning algorithm can imbue
Intelligence on even simple objects like
toys and jackets
finally I want to thank my brilliant
collaborators Eve Sal Kendra kin and the
Lansing sample thanks for coming to my
talk
foreign