- So this work was done with Anind Dey,
Chun Yu, and then our co-advice student, Orson Xu,
who unfortunately couldn't be here because of a visa issue.
So he's going to talk a little bit via a recorded video
about some of what we did, and then I'm gonna give the rest
of the talk, just 'cause we wanted
to make sure he still had a chance to be up on the screen,
and that's him right there.
So we'll se how this goes in terms of the audio quality.
- My topic is clench interaction,
novel biting input techniques.
- No, okay.
One sec.
I'm not worried about that.
I will do this instead.
Good, well, in actuality, I think
I could probably skip this slide
'cause we just saw such a wonderful motivation
for why bite interaction is something we might want
to consider, but I do wanna just touch
on some of the previous work that's been done in this realm.
So what you see here are some examples
of tongue and jaw-based input.
So in particular, on the left is a joystick
that you can control with your tongue.
Here, we can detect whether the user is applying pressure
to their cheek using the tongue.
And in this example, again, it's measuring the changes
in the shape of the face caused
by the tongue touching it from the inside.
So that's one set of examples of
how people have done interaction in and around the jaw.
And then the other thing that has been most commonly shown
in the literature is tooth click-related interaction.
So what we're seeing here, for example,
is an in-ear bone conduction microphone
that's being used to detect tooth click sounds.
And here is a confirmation technique
that's using tooth click in conjunction with eye gaze.
And then this is some work by Dan Ashbrook and others
that was also looking at which tooth was being clicked.
So five different pairs of teeth were supported.
So these are just some examples of some of the sensors
that have been used for this in the past.
Now, in our work, what we're actually using
is the clenching motion of the jaw,
so specifically pressure rather than tooth clicking.
And also looking at different regions of the mouth
that you could clench with.
And the main goal of this paper is really
to explore the design space for clench-based interaction.
So in particular in our work, we were interested in force
and specifically how many different levels
of force can somebody effectively control,
as well as what kinds of interactions make sense
in terms of repetitions, like double clicking
with your teeth, or more sustained interactions
like clenching and holding.
And then finally, we looked asymmetric,
meaning left versus right, and symmetric,
meaning clenching with the entire jaw
as aspects of the design space.
And in order to explore all of this,
we did a series of studies.
I'm gonna talk about some of them
in slightly more depth and some in less.
So we'll start with the first study,
which was just a basic target acquisition task
that was really meant to be both a proof of concept
and to give us basic parameters
for how to best signal the end
of an interaction as well of what levels
of force somebody could easily control.
And just to give you a picture of the sensor that we built.
Unfortunately it was not wireless.
But what we did was we encased a force sensor
in layers of material that would allow for in-mouth use,
and the whole thing is about 1.5 millimeters thick
to keep its, the interference
with regular jaw use to a minimum.
So that's the sensor we designed.
Although I will say that this, again,
the sensor was intended to prove
that this was a useful approach, and not necessarily
to be the final way that you would do this.
Okay, so this is an example of the interface
we used in the first studies,
so really a pure target acquisition task,
and in this case, the user has to select
from among three different levels.
We had four different conditions
in terms of the levels of force
that the user needed to control.
So here's the same example with eight different levels.
And in these examples, what you're seeing is
what we call quick release, which is to say
that when the user achieves the level
that they were aiming for, they just let the force go down
to zero by opening their mouth.
We also explored other techniques
for how to indicate the end of selection.
So for example, the dwell technique involves staying
in the region for a full second.
And we also, I'm not gonna talk about it today,
but you can look in the paper,
we did a version where a keyboard key was used,
what we called button, in order to end the selection.
And then we also explored two different forms
of visual feedback.
So the first is full feedback,
and you can see it here on the left.
And in the partial feedback case,
what really we were trying to aim at was
to get a sense of how all this would work
as a eyes-free display, but because it was a study
and we needed to indicate the target,
we do show a little bit of information to the user,
not where their cursor is, just where they're aiming
and when they get there at the end.
So that was the design for the first study that we did.
And just to talk about some of the results.
The first thing we saw was that
in the case of dwell selection,
there is a real impact of the number of levels on speed,
and I'll explain this in a second when we get to error.
This is speeds without the one-second dwell time,
just to give you a sense of how high these are.
And the y-axis there is seconds,
and then the x-axis is just showing the different levels
that needed to be distinguished.
When we switched to quick release,
this effect gets to be much smaller.
It's still there, but the impact of the number
of levels on speed is not as significant.
We also found that quick release was more accurate.
So what you're seeing here on the y-axis is
how many times the user exited the region
once they had entered it, that was needed for selection.
So you can see that in the case of dwell,
especially with eight different levels,
it was really hard to keep the target selected
for the length of time necessary
to complete the dwell selection.
And we believe that is a strong explanation
for why we saw such differences
in the target selection time in the dwell case.
Whereas with the quick release, on average,
they had less than one error per selection task.
So, that's just a more reliable method for doing things.
Okay, we also found that the visual feedback
doesn't have a really big impact on time,
which is a good signal that a heads-up variant
of this might be a reliable option for people.
And that's true even when we have eight different levels,
which is really encouraging in terms of the fidelity
with which someone could use this.
So that was the baseline study that we needed
in order to launch a more deeper investigation
into what are the kinds of interaction techniques
that somebody might want.
And by the way, the ends over here is just indicating
how many users participated in each of these studies.
So this second study, we wanted to scale up
and get input from a lot of people
about interaction techniques,
and we actually looked at 24 different techniques.
I will refer you to the paper for the details on that,
but they involve variations on not only the force levels
that you see in the first study,
but also whether an interaction
would involve a single press, a double press,
a hold and release kind of thing.
So we looked at time, and then also symmetry.
And the way that we executed this study,
was we just asked people to pretend they had a sensor
in their mouth, try each of the things out,
and then just rate it on preference.
So this was just purely trying
to help us narrow down the field in order
to do the final studies that let us explore
in more depth how this worked.
So here, you can see that when people experimented
with different levels of force,
they had a preference for light, and so anything
above three and a half here is a positive rating.
So all of these were okay, but light
on the bottom there was the most positive
on a seven-point liker scale.
And we asked about other things too,
but I'm just gonna talk about preference today.
When we add in single, double, and triple clenches
with the jaw, we see that people found it more comfortable
to do a single in their experimental simulation
than double, and more comfortable to do double than triple.
Now this was for symmetric actions.
The results for asymmetric actions,
meaning left versus right, are very similar
in terms of what is best, but you can see
that there's a slight reduction in preference
when they're doing it only on one side.
And when we look at sustained clench,
we had similar results, except that symmetric
and asymmetric were a little bit closer together.
So based on this, we selected
the six most preferred interaction techniques,
and then we started to develop applications
that were a little bit less abstract,
in order to explore how people might make use of clenching.
So the first study I'm gonna talk about was in a VR setting.
And here, we asked each person to perform all six
of the actions that were most preferred in this context,
and then we also compared this.
So in every case, the user had a head-mounted pointer on
that they could use for 2D positioning,
and they could either, in one baseline,
just use the head-mounted pointer with dwell for selection,
and in the other baseline we have them a handheld button
that they could press for selection.
So we had those two baselines,
and then in each of those three conditions,
they tried all six interaction techniques.
And I just wanna show what the VR setup looked like
in the case of the clench interaction.
So what you see on the left and the right are feedback
about how strongly they're clenching.
And then those two lines there just indicate
where we had the cutoff for light and heavy.
And in this experiment, we only ever had three levels
of force that we were sensing,
but we played around with other aspects
of the interaction techniques.
And then finally, the little red dot just indicates
where the head cursor is.
So I'm gonna show the six interaction techniques in video.
So the first one was just a single clench,
and the head mouse was used to move over the target,
and then the clench is used to indicate selection.
In the second task, they had to color the bottom square
to match the top square using a force level.
And so the three force levels map
onto green, yellow, and red.
So you can see here, that they're just using the force level
to select red, in this case.
And that black dot is just moving them onto the next task.
Now they're gonna select green with a lower force,
and then we're gonna have them select the yellow.
In this task, we were comparing their ability
to use double and single clench
to indicate two different responses.
So what they're doing here, is they're looking
at the equation, and they're clicking once,
or clenching once for yes and twice for no,
based on whether the equation is actually correct or not.
In task four, we are using a sustained clench,
and it's symmetric, meaning they don't have
to differentiate one side to the other,
as was everything I showed you so far.
And what they're doing here is they're starting the clench
to grab something, they're using the head mask to move,
and then when they release, it gets dropped.
So that's a way of moving something across the screen.
Here, we're starting to use asymmetric interaction.
So by clenching on either the right side or the left side,
they can control the direction that the blue dot is moving.
And then finally, in task six,
we're using a sustained asymmetric clench,
and that's controlling where this dot is moving
along the linear actuator here.
So those are the six tasks we tested,
and you can see that they vary
in how long it takes to complete them.
So these are longer completion times
than for the basic target selection task,
and some of these tasks are also more complicated.
That's the clench case.
When we add the other two baselines,
which did not involve clench interaction,
again, just the head mouse was dwell,
and then also, the head mouse was button click,
we can see that there is not a lot of difference
between these three conditions.
So the stars do indicate where there is difference.
But the effect sizes are small,
even where the difference is significant.
So the main result coming out of this study is
that clench interaction is no worse than other options.
It's also not shiningly better in our results here.
And we did look at the error rate as well,
and we see slightly larger differences here.
In three cases, clench is better or equal,
and in three cases, it's a little bit worse.
Our final study, we actually had people
on a stationary bike, and we had them use the clench
to answer, mute, or reject a phone call,
to switch to a previous or next song,
and to turn down and up the volume.
So we're just trying to move more and more
in the direction of realism here.
And here, we compare to standard methods
for doing the same things using either earphones
that had some actuation on them, or just the phone itself.
In this study, we see that clench is a lot faster
than what was found in the other conditions.
Okay, so a little bit about limitations.
I was gonna play a recorded video here again,
so I'm gonna flip through the videos,
'cause I don't have the slides up.
But the thickness of the sensor, of course,
is not negligible, and having a cord hanging
out of your mouth is not ideal,
so we really need the completion of the previous paper
in order to get around all of that.
Although it might also be possible
to explore other methods, like EMG sensing.
And we did double-check
on whether there was any medical consequences
of clenching a lot, and it looks like we're okay there,
but not necessarily people's preference to do clenching,
especially if it's-- (video starts playing)
Oops, sorry.
Especially if it is the case
that it is sustained or frequent.
Just trying to pull up the second slide here.
So in summary, our goal in this paper was really
to demonstrate both the feasibility of clench interaction
and some of the design parameters
that you might use when considering
how to design such an interface.
And we did find some useful effects
that low forces preferred, but a range
of forces are accessible to people, especially
if you're using quick release to indicate selection.
That the use of a sustained clench is feasible,
and we sort of played around, again,
with these different parameters
of how many times you click and where you click.
And I guess I will end there,
thanking Orson for all his work on this project,
as well as the other co-authors,
and see if there are any questions.
Yeah.
(audience applauding)
- [Organizer] So we'll have time
for five minutes of questions.
- Okay.
- [Zhang] Thank you for your presentation, nice work.
My name is Zhang Qiang from the office in Germany.
And regarding the realistic use during biking,
have you also considered
if accidental clenching might have been, for instance,
when you're quite exhausted, and just squeeze your jaws.
- That's a really good question.
We did not explore intent to interact,
which you might think about, there's lots
of reasons why you might clench
that are not accidental as well, right?
And so being able to distinguish
when somebody intends that
and when they don't is always an interesting challenge.
Not something we looked at in depth.
It would be interesting to see
if you could develop a sense of force profile,
so what should be ignored, for example,
or whether there's a way that you'd want
to indicate start of interaction.
- [Zhang] Okay, thanks.
- [Audience Member] That was a really nice talk, thanks.
I was just wondering about the mapping function
between the actual clench forces and the levels,
because the levels themselves were evenly spaced,
whereas I would imagine, and I could be wrong,
that the force exerted wouldn't necessarily
be linearly actuated. - Yeah.
- [Audience Member] And then the second question was
for something that would be,
what about individual differences?
Did you notice the need for--
- Could you speak up on the second question?
- [Audience Member] Did you notice the need
for individual differences, to calibrate
for individual differences, in order to achieve accuracy?
- Right, so we did do some calibration
in order to determine what was the minimum and maximum force
that we wanted to map the range into.
I don't believe that we also used a variable force profile
in doing that mapping.
And unfortunately, Orson really should be here
to answer your question about
to what degree we did individual calibration in the study.
I don't believe we calibrated separately for each person.
I think we were able to determine a reasonable choice
that worked for everyone, but I would have
to refer back to him to be positive about that.
- [Audience Member] Okay, thank you very much.
- Yeah.
(whispering)
- [Thomas] Hello, Thomas Vega from the media lab.
I was wondering if participants with a overbite
or underbite experienced trouble
in performing the clench test, or?
- Thanks for your question.
So I should have been repeating the questions all along.
The question is about an overbite versus an underbite.
That was not something that we explored
in our study or had an issue with.
- I will also, I will ask a question.
- Sure.
- Okay, so you know how some people,
when they are distressed, they clench?
Have you considered to measure the stress levels
to calibrate according to that?
- That's a really interesting, actually, separate idea,
I think in terms of also how to use the system.
Because if you wanted to study stress,
I mean, we talked about galvanic skin response
and other things, but it would be fascinating,
I think, to just have a passively sensing clench setup,
and see whether there was interesting signals in there
that we don't get though other means.
It's, again, not something we explored,
but thank you for the idea.