- Thanks for the introduction Mike.
My name is Yang Zhang.
I'm a fourth year PhD student
from Carnegie Mellon University.
Today, I'm really excited to be here presenting our work
on Vibrosight, long-range vibrometry
for smart environment sensing.
This work was done with my colleague Gierad Laput
and my advisor, Chris Harrison.
As we just heard from Gierad in his talk
that people have envisioned smart environments for decades,
where our environments cannot only passively
take commands from us but also understand the context
and proactively adapt to it.
This vision relies on a robot sensing method
to detect human activities such as chopping vegetables,
writing on white boards and doing exercises.
As a result, much effort has been spent on
activity sensing in previous work.
For example, the most common approach uses camera
with computer vision technologies.
There are also acoustic approaches,
which deliver just the sound resulted from the activity.
The less common approach uses radio frequency
and capacitive sensing.
And finally, we can also attach sensor tags
to objects for usage detection.
These sensor tags can also be used to detect vibration,
which is the by-product of many user activities.
In Vibrosight, we also detect vibrations,
but uniquely at a long distance.
From this previous work,
we found several ideal sensing properties
for activity sensing, which Vibrosight tries to achieve.
First, we want to preform sensing
from a single sensing point
where the sensor can be easily set up
and placed out of a user's sight.
For example, higher up on the ceiling.
We also want to achieve simultaneous activity detection,
which was proved challenging in previous work.
And finally, we want our sensor to be privacy-preserving,
low cost, and high accuracy.
Now let me talk about our sensing principle.
We were inspired by the optical vibrometry
invented by Leon Theremin in 1950's.
This technique reconstructs vibrations from afar
using infrared light and later used by the KGB
in their eavesdropping devices.
In this project, we did a modern twist on an old technology
and the four very different application domain.
Here is an illustration of how optical vibrometry works.
We first have an emitter which bounce light off of a distant
and reflective surface, in this example a window.
Vibrations off surface caused by the speed,
or in our application, appliances running,
alters the light path.
We can place a receiver on the return light path
to capture the light intensity.
This signal can be used to reconstruct the vibration.
Relying on existing reflective surfaces require
calibrations between the emitter and the receiver.
We move the emitter or
when we want to measure different objects,
the receiver needs to be repositioned.
Ideally, we want to put them in one small package
and works with any angle and distances.
There are two unique ingredients
that allow Vibrosight to achieve this.
First is the use of retro-reflective tag as we see here.
They are cheap, durable,
and can be placed on a wide range of surfaces.
The second ingredients is a steerable mirror
controlled by two separate motors.
And here you can see the vibration signal
detected by our sensor shown on the laptop.
And the tag as tagged here,
as you can see there's a sensing laser dot on it.
Before building our systems,
we first want to verify the sensing principle.
I'm going to quickly go through the experiments that we did.
We first did a test on reflector material.
We tested different materials including a mirror,
a glass, reflective spray, corner reflector
and retroreflective tapes.
And we configured a actuator to output a sweep frequency
and measure the reflective light
with a photo diode sampled at 5 kilohertz.
Here we can see the spectrums.
We found that the retroreflective tape works as good
as the mirror and the glass.
We then did a test on the laser wavelengths.
So the three tested colors,
green, red, and the infrared work equally well.
So we decided to use the red laser
for the ease of debugging,
but ideally we want to use infrared in real world usage
because they are invisible
and thus it's unobtrusive in the environment.
Since Vibrosight uses optical sensing,
it might be affected by the environmental lighting.
To investigate this, we conducted a test
with four lighting conditions,
dark, daylight, incandescent light, and fluorescent light.
We found incandescent light induces 120 hertz noise,
and the fluorescent light induces more of a wide band noise.
For this reason, we decided to include filters
on our analog front end when we designed the PCB.
We also wanted to know how angle and distance
affect the signal, so we fixed the actuator
to output the frequency to one kilohertz and used our sensor
to measure the vibration signal amplitude.
Here is the result when we vary the laser angle.
As you can see here, there's a significant drop
between zero degree, which is when the laser
is being perpendicular to the tag, and five degree.
But still we can sense reliably up to 45 degrees.
When we vary the distance,
these results show that we can pick up the vibration
up to 16 meters.
We also did a test combining the two factors
to investigate if there are any interaction effects,
and we found it's just a linear combination of the two.
Now let's show a demo.
I would like to invite my assistant Mr. Gierad here
to help me. (audience laughing)
One second.
Since the board doesn't like to cooperate today,
let me give it a further de-bug.
Let's see.
Okay, let's put it on the ground.
There seems to be a disconnection of the wire, sorry,
so there probably won't be a live demo today,
but I will show you guys a demo later on.
(audience applauding)
Thank you Gierad.
Let me talk about the implementation.
The basic sensor unit consists of three part.
The first one is the three miniwatt laser dial.
It's labeled as class three A,
which is, I say, for short period of direct exposure.
And the second part is a PIN photo diode,
which converts the photon into a current.
And the third part is the bandpass optic filter,
which help us to improve SNR.
We built a sensor board to measure signals
from the sensor unit, so please find the more details
on this board in the paper.
As I already mentioned, we constructed
a steerable mirror platform.
As you can see, there's an illustration on the left part,
on the left of the screen that shows a complete sensor.
Our board's firmware controls the motor and measure
the reflected laser signal at five kilohertz.
We used the FFT result as the features
to train two label random forest classfier per object
for vibration on and off detection.
The result is fed into applications.
Here is a video that shows
a complete overview of the system.
So as you can see, the user first tags
the objects of interest.
Then our sensor scans across a whole room
to find these tags based on the reflected light intensity.
Here as you can see, the laser is being scanned
across the room and the detected tag coordinates
are shown on the laptop.
So once these tags our found,
our sensor cycles through them
in a round robin manner.
It collects one second data from each tag
for vibration sensing.
I just turned on the coffee machine over there
and the detection result is shown on the screen.
Here the microwave is running
and it's detected on the screen, it's shown.
So right now three things are happening.
Chopping, coffee machine, and microwave running.
So all picked up by the sensor.
Now let me talk about the evaluation.
We did the study at four locations,
as is shown here.
There's a kitchen, a gym, an office, and a workshop.
We picked six objects in each location
and attached five centimeters square,
with the reflective tags to them.
We then run an object usage detection study.
So on the left, you can see the vibration spectrum
of all tested objects when they are active.
To test the vibration interference,
we collected the data at four conditions.
First when no object was active,
then when one object was active,
when two objects were active with all combinations,
and finally, when all objects were active.
In each condition, we collected three seconds of data
each time the laser land on an object.
We collected 10 rounds of data
which yields roughly 16,000 data points in total.
We used leave one round out validation
to avoid overfeeding, and the result shows
an object usage detection accuracy of 98.4%.
Here, we break down the accuracies
into the four data collection conditions.
As one might expect, accuracy is the highest
when you just turn on one object,
but we can see that even when multiple appliances
are active, Vibrosight still achieved high accuracies.
We also tested our system one week later.
This time the data collected was only used for testing,
so using the model we trained one week before.
The result shows an accuracy of 96.2%,
which was not significant different,
suggesting robots needs over time.
Three seconds of sampling durations of on time,
especially there are many objects in a room
that we are cycling between.
We want to investigate shorter sampling durations.
We found that we can use the duration of a half a second
without sacrificing the accuracy.
It would also be nice if Vibrosight
could also automatically recognize objects
without user labeling.
So to see if vibration alone
is distinctive enough to recognize this,
we built a 24 label classifier
with the data from all objects we collected from the study.
Leave one round out validation shows an accuracy of 92.1%.
I want to conclude my talk
with some supplemental uses of Vibrosight.
First, human movements inevitably results in vibrations
in the environment.
Here we can see the signal from a user
sitting down on the couch.
Similarly, Vibrosight could monitor sleep restfulness
if the tag is placed on the headboard or a bed frame.
Vibrations resulting from knocking or typing
on a tactile surface are also very apparent to Vibrosight.
Thus, this is also possible to support some basic inputs
such as double knocking to create a notification.
For devices with speakers, motors, or other actuators,
it is possible to incult information
in the form of vibrations.
This opens a very interesting communication channel
for richer applications.
For example, devices could emit their brand, model number,
or IP address to expose additional functionalities.
And finally, Vibrosight can detect small and movable objects
by sensing the surface they are sitting on,
such as a kitchen countertop
or a work surface.
Vibrations resulting from these small objects
shakes the surface and thus can be detected by our system.
Of course Vibrosight also has
several significant limitations.
First of all, we require line of sight,
which means that if the sensing beam is blocked
by the user, we cannot detect vibration.
And second limitation is the cycling speed.
Since we have a round robin looping mechanism,
having more appliances in a room
will lower the refresh rate.
And finally, the mechanical steerable mirror
might not be as durable as mirror,
which is more compact and has a faster switching.
We have open sourced this product
so you can find all the source materials in our gig hub,
including the data we collected in the user study.
Source called for the software, firmware, 3D files,
and the PCB design files.
And finally, I'm happy to take questions.
(audience applauding)
- The live demo was also working right before the talk.
Of course it broke.
Any questions?
- [Daniel] Daniel Ashbrook, University of Copenhagen.
This is really fun work.
I'm wondering if you have any thoughts on
the trade off between the amount of time it takes
to sense each object
and the number of objects you have in the room.
So if I wanna really put a sensor on everything in the room,
I might have a thousand tags
and it could take a couple hours to scan the entire room.
- Yes.
So in the study we found this quite interesting.
So actually as I mentioned we investigated
the sampling duration.
We found for some objects that have
constant vibrations patterns,
for example like microwave running,
the dishwasher running,
you can just pick up their vibration
if the laser stay there for only 0.1 second.
But for other appliances, for example phone ringing,
or I'm using a hammer,
these might require longer sampling.
So actually we can use adapt duration, sampling duration,
to improve this refresh rate.
Or, another idea is if you have two lasers,
so you can basically double your sampling.
So actually in future work we want to investigate
what if I have for example the random dot projector,
we have a millions of random dots
projected into the environment.
In that case, the refresh rate won't be a problem
because we're just sampling everything,
what is in the entire room at the same time.
- [Daniel] Okay cool.
So one other question is how do users
actually indicate what these objects are?
So again, say I have 1,000 objects.
We have about 30 objects.
How do I say this is the dishwasher,
this is the coffee maker, and so on?
- Yes.
The first solution for that
is whenever user, they'll tag the surface, right?
After our system detects where are the tags,
we can turn on the visible light,
point at a specific tag, ask the user what is that?
And the user label that appliances in that scenario.
In another scenario we can automatically detect,
recognize the vibration pattern of the objects
the first time they are used
based on the vibration signature.
- [Daniel] Thanks.
- [Announcer] Okay let's thank the speaker.
(audience applauding)