- From UBC, and I'm here to talk about how we can get
crowdsourced feedback about vibrotactile sensations
and, more generally, haptic sensations too.
I'd like to start with a question.
How many people here have a device that buzzes?
Pretty much everyone, right?
How many buzzes do those devices make?
One, yeah, exactly, not very many.
And this is the thing.
With haptics devices, especially these low-fidelity
or these vibrotactile devices,
you really have to design these different sensations
to make them effective.
When you do, though, you can do lots of different things.
So we're gonna talk about vibrotactile sensations.
Those are vibrations that you feel in the skin.
When you do design them, you can make a lot of cool things.
You can tell someone that they just beat
their personal record or to run faster
to catch up with a ghost if they're exercising.
You can communicate emotion,
adding a personal touch to your communication with others.
Or you might be able to invisibly tell someone
when it's time to move to the next slide.
But doing this, of course, requires a design process,
and haptics remains very challenging to design.
One of the big obstacles is getting feedback.
So if you're a designer and you've created some vibrations,
you might bring some people into the lab
to get some feedback and see if they're working,
it's actually telling someone to move to the next slide.
Now, this is, running in lab studies can be very slow.
It can be very expensive.
And we've seen a lot of work
coming out of other types of design
about how to do crowdsourcing
and get lots of people running studies very quickly,
and this is what we want to do.
If you look at other modalities,
you can find that you can get feedback, for example,
on graphic design, so whether using contrast,
proximity, color effectively.
You can generate ideas from the crowds.
You can come up with a message for a birthday card
or something similar, user stories.
You can even run perceptual studies at scale
as long as they're visual,
which allows you to collect lots of participate data
very, very quickly and very affordably,
so this is very attractive.
So what do we do to go from here to here?
What can we do to actually crowdsource haptics?
Well, there's been a lot of work on how to try
and add haptics to broadcast media,
such as adding it to a movie or a video.
So tactile movies here, whenever, I think it's Spider-Man,
is jumping from one wall to the other,
you can feel that motion on these gloves.
Now, the problem is I don't have a pair of these gloves.
Does anyone here have a pair of those gloves?
One person, okay, one or two,
so we have a crowd of two.
So this is a little bit limiting,
and, of course, we can't ship out these devices to everyone
because then it's slow and expensive
and all the things that we're trying to fix.
And this is fairly consistent when it comes to haptics
in other media.
There's lots of different approaches,
adding the equivalent of camera motion to haptics,
so haptic cinematography.
When something tilts, you kind of feel tilted.
Or adding something to YouTube.
We can do that in the software,
but at the end of the day,
we have custom or high-fidelity hardware
that we just can't send out.
Not everyone has.
So this is challenging.
We can't send high-fidelity haptic devices,
so the question that we asked is,
"What can we send instead?"
You can send lots of things over the Internet to the crowd.
You can send graphics.
You can make someone's phone vibrate.
You can send audio.
And the key insight that we have here is that
could these be used as proxies to stand in
for the high-fidelity haptic sensation,
and that's the idea behind HapTurk.
You take a high-fidelity haptic sensation.
In this case, we're gonna tall about vibrations.
You translate it into proxy vibrations.
You send them over a crowdsourcing tool,
like Mechanical Turk,
and then you use that feedback to inform your design.
So let's put this is an official research question.
Can proxy modalities represent
perceptual qualities of high-fidelity haptic vibrations
to the crowd?
Let's break this apart.
We're gonna start with perceptual qualities.
That's the beginning of our journey.
So if you look at the literature,
there's lots of different dimensions
to how we perceive touch.
They're still being developed.
Some of the major ones are duration, energy, and speed,
which are very low level physical cell qualities
that contribute to our perception of vibration.
There are other ones like roughness, urgency,
and pleasantness, which are a little more
emotional, affective, and high level perceptual in nature.
Now, most of these found using perceptual studies.
Some of them are rating scales.
Some of them are found through clustering
and multidimensional scaling.
Others are tags that people can use to describe vibrations.
We standardize in using a slider from zero to 100
to keep it very easy for our participants in the crowd
to be able to write these things.
All right, so we have perceptual qualities.
The next thing we need are some vibrations
to actually represent, and that's where we went next.
We used a C2 tactor.
I have one right here.
You're welcome to come and find me later
if you want to see what it feels like.
This is like a tiny speaker.
It's a voice coil type actuator,
so it controls amplitude and frequency.
We use it together with tool that we've developed in our lab
called VibViz.
Now, this is a visualization of 120 vibrations
along different perceptual facets,
and so it allowed us to come up with a diverse set
of 10 different vibrations that we label V1 through 10.
We chose these to be representative of this wide space
but also diverse in representing these perceptual qualities.
So we have our vibrations.
The next thing we need are our proxy modalities,
and that's where we went next.
So if you remember, the question that we asked is
"What can we send?"
Well, the easiest thing that we might be able to send
is graphics, so let's start there.
And, actually, we came up with two different visualizations
to represent vibrations.
The first is kind of the default, a waveform.
When people worked with VibViz,
what we found is that they would use
these waveform visualizations to skim or find similar
or different vibrations,
so we thought this was a good starting point
as a status quo, but it's the default.
Can we do better?
And so we decided to try and design a new one.
We began by exploring different shapes and looking at
texture and color to try and see different ways
that we might be able to represent a vibration.
Eventually, we refine through piloting.
We eliminated color because it was emotionally charged,
and we eventually had three different candidates
that we then kind of had to show off in a user study.
Finally, we end up with this line-based system.
We have a key of how to translate a vibration
into this system.
The roughness of the line corresponds to the roughness
that you feel in the vibration.
The weight and height of the line
correspond to the energy,
and we have a measure for how long and short
the vibration actually is.
So those are our visual proxies.
We've got a waveform, and we've got design.
We tried to keep them relatively similar,
so they're comparable.
And, of course, now that we have a visualization,
I can show you all our vibrations, and there they are.
Of course, you can consult the paper
to see them in finer detail.
So that's trying to send graphics.
The next thing that we might try to send is vibrations.
If you're sending a graphic, it's very easy to send,
but it's rather indirect, whereas vibrations,
feeling it in your phone,
that's a much more direct translation
of these high-fidelity haptics.
We did consider going with audio
or some of these other techniques,
but combining our expertise,
we didn't have an audio expert on the team,
and, also, the directness,
we decided to go with vibrations first.
And to do this, we translated each of our
high-fidelity vibrations into phone vibrations.
Now, your phone has a very limited actuator.
It's an eccentric mass, so all it does is
kind of shake or not shake with different voltages.
And if you're controlling it through something like Android,
which is what we did, you're even more limited.
You can only send bursts.
You can say vibrate for 15 milliseconds,
40 milliseconds, et cetera,
and so we went through this process of piloting
and translating these different vibrations into the phone,
and we had sort of this pulse-width modulation approach
where you have shorter and shorter bursts to make it weaker.
We really had to play with representing energy
and rhythm effectively.
So now we've got our proxy modalities.
So right now we basically got everything we need
to start looking to validate this approach.
Can we actually do this crowd sourcing thing?
To actually validate this,
we came up with a couple of main goals.
The first is we need to collect ground truth ratings
of our high-fidelity vibrations.
The second thing we need to do was compare these proxies
to those ratings, keeping everything else the same,
so doing an in-lab study.
Finally, we'd actually want to deploy it
over Mechanical Turk to see if anything breaks down.
We organize this into two different studies,
a local study to collect the ground truth
and compare to it and then a remote study
to test whether anything breaks down when we go remote.
So let's go to the local study next.
We had two different between-subjects conditions
based on the proxies,
so one was the low-fidelity vibration proxy.
The other was the visualization proxy.
We had people rate both the high-fidelity vibrations
and the proxy so that we could try and reduce
some individual differences,
and they did this for every vibration
for both of the modalities
or all three in the visualization case.
And then what we did is we compared the ratings
of the proxies to the corresponding ratings
for the reference vibration.
Now, if you ran a t-test or an ANOVA,
you'd be able to tell if they were different,
but we wanted to show that things were the same.
And if anyone knows a lot about stats,
you know you can't try and prove the null hypothesis,
so instead we employed something called equivalence testing,
where you set a threshold level, in this case a delta.
We use the standard error of the mean
to try and set a, at least a,
try and keep it related to the amount of error
in the system of the rating system.
And we found whether things could be equivalent,
whether we couldn't tell if they were equivalent
or different or whether they were different,
and this is what we used in our analysis.
Here are our results.
There's a lot going on, so we're gonna zoom in in a second.
Along the top, we have a perceptual quality,
so speed, duration, energy, and then et cetera.
And then on the side, we have our different
proxy modalities, so the waveform visualization,
the design visualization,
and then the low-fidelity vibrations.
So let's zoom in.
These are the pleasantness ratings
for all 10 vibrations for the waveform visualization.
You can see that the colors,
we're gonna focus on the colors right now, not the location,
'cause that's what actually tells us what's valid.
So vibration one was well-represented.
It was equivalent.
Vibration two, we couldn't quite tell.
It was uncertain.
And then red for several other vibrations
meant that they were different,
so pleasantness was, for most of these
or for half these vibrations,
was not well-represented using this visualization approach.
So let's zoom out and let's use that
to try and look at this data.
Now, the first thing you notice is that
there's a lot going on here.
Some vibrations were well-represented, some weren't,
so let's see if we can find some trends.
The first thing is that duration and pleasantness
seem to be the most translatable.
If you look at the duration column,
there's lots of green, and most of the vibrations
had at least one modality that accurately
represented the ratings.
With pleasantness, we had something similar.
There's more uncertainty,
but it comparatively was a bit better.
With speed and urgency, we found that sending
direct vibrations seems to be a bit better.
There's at least less difference
and more equivalence going on there,
so that means that sending something direct can be helpful.
But in other cases, it seems like visualization
might be a bit better too,
and so we have this three-way interaction effectively of,
it depends on the vibration, it depends on the modality,
and depends on the perceptual quality.
That's challenging.
Especially challenging was energy.
There's lots of red in that column.
If I had to pick, I'd go with vibrations,
but this is something we can focus on in the future.
The other thing to notice is that the designed visualization
tends to work quite a bit better than the waveform one,
and so by creating new visualizations,
we can actually improve this process.
This is not unexpected, but it's encouraging to say,
"Hey, if we design new visualizations,
"there's a value in that."
So to wrap up, in study one, we saw that
some of these translations succeeded,
and that's encouraging.
We saw that we can do better,
and we have some ideas of where to go from there.
But before we start to try and do better,
we wanted to deploy online and see if anything breaks down
with Mechanical Turk,
and that's where we went next, to the remote study.
So, remember the structure of the local study.
Study two was very similar
except we could only deploy proxies,
and we ran similar levels of participant numbers
so that it was comparable error rates, et cetera.
Wanted to keep it as similar as possible to the local study.
And they're compared to the same reference points,
trying, again, keep things as similar as possible.
Here are the results from that study.
The first thing to note, and it's hard to tell right now,
but there are more details on the paper.
On the whole, it's similar results,
and if we highlight the ones where they differ,
there's only about 10% of the vibrations
actually disagree between the remote and the local proxies.
If you can compare proxy to proxy,
only in these cases does it actually change
whether it was equivalent or not.
So that's great.
So 88% or ratings were consistent with local proxies.
We did see that some participants were unreliable
in the vibration case,
so they said that they are using Android,
which is the platform we use for vibration,
but they were actually on Mac or Windows,
so they're clearly lying.
We didn't include that in the analysis, by the way.
We gathered more participants.
Great, so what are the takeaways from this?
Let's go back to our first question.
Can proxy modalities represent perceptual qualities
of high-fidelity vibrations to the crowd?
Yes, for some proxies, qualities, and vibrations,
and it depends on the combination.
To give more detail in the conclusion,
we believe that proxies can be viable.
There's some evidence of successful proxies.
More importantly, we found what worked
and what can be improved
and have ideas of what to do next
and found that there were no major breakdowns
when you went to the remote Mechanical Turk situation.
So for future work,
we're gonna try and improve upon these proxies.
We're gonna try and combine them together,
see if you can triangulate by studying a vibration
and a waveform and a visualization.
Maybe try and animate them or use sound,
and, of course, automatic translation's gonna be
very important to employ this practically.
Right now we hand-translated everything
into approximate proxy modalities.
That won't scale,
so we have to actually generate some tools.
We also want to look into how people get feedback.
Here we looked at a very strict case of equivalence,
whether the rating scales themselves were very close.
But if we ran something like ANOVAs
or t-tests or tags, we might be able to get
useful information that's more consistent
despite some of the noise of these rating scales.
And, finally, we do want to collect more data,
run a bigger study, and try employing best practice
for Mechanical Turk.
We were fairly purposefully naive.
We wanted to see anything that participants would do,
like lie or whatever, but now I think we can start to
structure our feedback a little bit better.
One thing I want to note is that this is in line
with a larger trend of work that we've been doing
in our group on online haptics.
We have VibViz, which you saw,
which is free and available right now.
It's an online collection of visualized vibrations.
We also have an online haptic editor
that you can use right now,
and I've got things that vibrate
that you can plug into your laptop
to try these out if you'd like,
so come find me if you want.
Thank you very much, thank you to my co-authors here,
and let me know if you have any questions.
(audience applause)
- [Audience Member] There's time for a question.
Please state your association.
- [Paul] Hi, my name's Paul.
I'm at Copenhagen University.
So let's say you've done your follow-up study
and you're really happy with your design
and you know that the results you get
are kind of what you want to get,
what kind of study would you run with this system?
Because you're using super high precision actuators,
so I would think you'd be investigating very subtle effects.
And what do you see yourself using this tool for?
- Right, so the case that I would imagine is,
if you think back to those use cases we show
at the beginning, let's say the presentation one
'cause I'm giving a presentation.
You might have a couple of different ideas
of sets of icons to tell people to move to the next slide
or speed up or slow down, et cetera.
You might set that cluster out to get people to say,
"Oh, this set is better,"
or, "I like this vibration more than this one,"
or, "This one's more salient."
So I think trying to come up with the right design language
for your haptics is where we've been thinking.
You might be able to do some perceptual studies as well.
We found it very difficult with all the noise
in these rating scales to find out
where other rating scales might be
and what the perceptual dimensions of haptics were,
and we're hoping to scale those up as well
to see if there's even higher level things,
like emotional content.
Does that help?
Okay, cool.
- [Paul] Yes, that answered my question.
- Great, thank you very much, we can follow up later.
- [Pablo] Hello, this is Pablo Perez, Stanford University.
So can you explain how do you choose the sigma
for the equivalence testing?
Because that's not a trivial parameter.
You really need a lot of prior information to pick.
And, also, did you make any type of correction?
Because you ran a bunch of studies.
Do you do any type of statistical correction on it?
- Yeah, yeah, so we use Bonferroni correction
for multiple comparisons, and then,
so typically you're right.
In equivalence testing, you have some sort of
practical threshold derived from the field.
We didn't have this right now.
What we did instead is we tried to just stay
within the standard error
of the ratings that we collected,
so it's tied to the noise in the system
of the ratings themselves,
which was about 15 to 20 on that 100-point scale,
which means that we can, I think if things were equivalent,
I would say they probably map towards
a five-point Likert scale,
so we can get some reasonable psychological results
from some of these, but you're right.
There's a lot of noise in these rating scales,
and this is actually part of the reason
why we really want to crowdsource this,
so that we can start to find out where these means are.
- Okay, all right, thanks. - Yeah, thank you.
(audience member speaks off microphone)