- Good morning, everybody.
I'm SunJun Kim from Kye-St,
and we're supervised by Kee-Yong-Dee
from HT-Shell Lap-Kyste.
I'm now introducing you TapBoard 2
which is the combined
pointing and typing interaction surface keyboard.
The motivation of this work is the burdensome
of the switching between the keyboard and mouse
or between keyboard and touchpad.
And of course there are many solutions
as many attempts have been done
to combine typing and pointing in the same space.
These works (mumbles) pointing functionality
from physical keyboards and (mumbles) in various ways.
During this work, our previous work named TapBoard
at CHI 2013, which is a (mumbles)
only tapping as the keystroke and ignore others
for other purposes.
The other purposes meaning, at CHI 2013 was,
some kind of (mumbles).
In this case, TapBoard2,
which uses non-tapping gestures for mouse manipulations
were precisely non-tapping gestures in this case,
meaning dwelling or dragging gestures.
We are visiting each interaction one-by-one.
Of course keystrokes are done by taps
and dwelling and dragging
used for manipulating mouse cursors.
This is the basic concept and we implemented
two different types of the mouse clicking gesture.
One is named AddFinger gesture,
which is done by adding one tap
while manipulating the cursor.
Another one is named Tap gesture
which is consecutive tap after releasing
the manipulating finger.
So those two gestures
mimicking touchpad interaction.
The former one clicking (mumbles) button
and the latter one is the traditional tap-to-click gesture.
And to support some additional interaction,
to support more regular mouse operations such as
the drag and drop.
The other was button press and hold.
Additional finger dwell during cursor manipulation
acting button press and hold,
and two fingers scrolling
working as a mouse (mumbles) scroll
or scroll gestures.
This gesture may mix with regular touchpad operations
so we expect the user to adapt this concept
very easily and quickly,
and more about these gestures
designed to be corrective with typing impression
without any conflict
so without any more switching.
And we first attempted to implement this idea
with a finite state machine.
However, the (mumbles) gesture increases (mumbles)
and transitions between states are (mumbles)
so that we need to define some more systematic approaches
to implementing algorithms to detect the gestures.
So we found that most of the transitions
in our finite state machine
happens when the finger count changes
and when dwell time lapsed
or finger (mumbles) some moving distance.
Interestingly, most of the previous work
on gesture on touchscreen keyboards
also relies on similar actions.
Therefore, our algorithms can be generally applicable
for those applications.
Given those three features,
we developed a simple and effective
gesture recognition system as follows.
We'll visit them one by one.
The first item is defining touch phases for each touchpoint,
and this definition takes moving distance
and dwelling time into account.
In TapBoard system, a touch point may have
one of these three phases
which is touched, dwelling, and moving.
Every touch point starts with the touched phase,
and after a certain amount of time
it goes to the dwelling phase,
in this case 450 milliseconds from our previous work.
And from those two states,
after the cursor moves a certain distance,
in this case 5 millimeters,
it goes to moving phase
and moving phase is the final phase
until (mumbles).
Okay, now we've defined
touch phases and move on to system state.
A system, in other words a multi touch device,
may have multiple touch points
having different touch phases.
And the system enumerates the number of touch points
in different phases,
represented as three numbers.
And these three numbers indicate the so called system state.
Now we move on to define gesture,
which is defined as
a sequence of the system state.
For example,
when the user interacts with the system,
the system changes states over time.
For example, user lands the finger and then lifts up.
Then these three numbers,
changing history of the system state
and define the tap gesture.
For another example,
user lands on and slides and lifts up.
And this is defined as a drag gesture,
those four numbers.
So, with this system we could implement
proper interaction with that of number sets
like on the right side,
with (mumbles).
And we evaluate those design interactions
and gesture recognition systems
with the following setup.
The system consists of one Microsoft surface tablet.
And the tablet generator, Legacy USB keyboard and mouse,
signal to the experiment a variation system.
We prepared three conditions.
The first one is the CA
which means the Combined AddFinger gesture interaction,
and CT for Combined Tap gesture interaction.
And as a baseline, ST means the Separated Tap
which is two devices.
One for keyboard and one for touchpad.
Let's look at each condition in more detail.
Combined AddFinger condition used the left hand to click,
and the Combined Tap condition used a release and tap
strategy for click gesture,
and Separated, ST condition uses two separate devices
and the user needs to switch between two devices.
We prepared 18 participants
which is gender equalized
and we designed a within-subject design,
and each participant tried 243 trials
for each participant.
Each participant tested three conditions,
all of the three conditions.
And the task including pointing and typing was designed.
The pointing task was the (mumbles) pointing task.
After that you then need to type a short word
that consists of five to seven characters
from Enron corpus.
So basically this is a pointing and typing (mumbles).
We'll briefly report the key findings,
so for more details please check our paper.
CA condition, Combined AddFinger condition,
(mumbles) interaction,
is having a more
pair of pretrial throughput,
and combined condition discriminated
pointing and typing with more than 96 percent accuracy.
However, some discrimination (mumbles)
combined condition's typing speed
about two WPM.
And more interestingly,
most participants,
so 15 out of 18 participants liked CA the most,
and CT for two participants and only one participant
liked the ST, the baseline condition, the most.
For our application we may apply this approach
to (mumbles) and (mumbles) like HP Sprout,
which is the (mumbles).
And for regular touchscreen keyboards
to simulate the mouse operation with a multi-touch surface
we can expand this work to the physical cable
to combining the typing and pointing integration.
Okay, to conclude,
I introduced TouchPad-like TapBoard 2 interaction
and evaluated
with higher accuracy and high user preference.
So, I'm happy to take your questions.
Thank you.
(applause)
- [Voiceover] So, the crosstalk between typing and pointing
is always a confusion, of course.
Back with J Point there was,
and I did some in-keyboard pointing device work also,
but you say that there's a 450 millisecond dwell
that you required to start,
and that's an awful long time.
What do you think about that dwell,
the half of a second dwell that you need
to start recognizing?
- That threshold actually came from our previous research
at CHI 2013,
and participants are asked to rest
and typing (mumbles),
in other words, dwell and typing,
we found that threshold from those data.
- [Dan] Hi, Dan (mumbles) Rochester Institute of Technology.
You mentioned implementing it on a physical keyboard
and I'm curious how you would be able to do that.
- Actually, we need more state,
for example touch and press state
for application and physical state
so we need to re-implement the system,
redesign the system to
application on a physical device.
So this might be our future work.
- [Dan] What about the actual sensing of the touch?
- Excuse me?
- [Dan] The sensing, how would you sense the touch
on the keyboard?
- There are a number of implementations
such as the touch display keyboard
that senses the keyboard,
and we, in our previous work,
we developed the keyboard tapping electrode
to sense the touch.
Or we can implement the (mumbles)
beneath the cable to detect
where we're moving and touching.
- [Dan] Cool, thank you.
- [Voiceover] All right, this concludes the session.
Let's thank all the speakers again.
(applause)