- Thanks for the introduction.
My name is Poorna, I'm from the University of Notre Dame.
Today I'll be presenting a paper on
Replicating User-defined Gestures for Text Editing
on tablet devices.
And I'll begin by asking you all a few quick questions.
So, how many of you use touchscreen devices regularly?
We all do, right?
And, how many of you use these devices
for text entry and editing?
Be sending emails, messaging.
And how many of you use a stylus regularly
with these devices?
Okay.
And how many of you regularly write on paper
with pen and pencil?
Okay.
Thank you.
So, I ask you these questions simply
to make the stock model of you
and your responses are actually very similar
to the ones we got
from our own participants in the study.
So what's text editing like on these devices?
It can be cumbersome, right?
You first have to position the cursor with your finger
and then widgets or carets appear using
which you can select text mode precisely.
And then usually, text editing involves two steps
on these devices.
You first have to position or select
and then choose the action to perform.
And given the advances in stylus technology
and improved digital handwriting recognition,
handwriting can actually
be considered a promising alternative
and it combines the ease of writing as well as
and afford features of digital document.
And we have these vision of an ideal handwriting application
that enables handwriting and recognition anywhere
on the document and we shared this vision
with researchers not only in the present,
but also researchers, like, three decades ago
when tablet devices had just been introduced
and they didn't, they lacked a lot
of the capabilities which are available on today's devices.
However, like despite our modern devices being very capable,
our current handwriting applications
are not really intuitive.
For example, you have to write in like designated boxes,
which then gets converted to type text,
but they're getting better.
Like for instance, this is MyScript Nebo app,
which actually lets you convert handwriting
by simply tapping on it.
And this paper actually focuses
on devising intuitive gestures for text editing
for such handwriting applications.
And to do that, we draw inspiration from
and we replicated gesture-elicitation studies
from papers that did this work like 30 years ago.
And those were like by Wolf and Morrel-Samuels
in 1987 and Welbourn and Whitrow in 1988.
So what is gesture elicitation?
So we elicit gestures from participants,
that is we give them, we give them the results
of an action or task known as referents
and we asked them to come up with gesture
that would result or produce that referent.
And for example, this is the referent
for the insert word task that we used in our study.
So we gave them, we told them what the result would be
and we asked them for gestures they would perform
that would result in that referent.
So our goals was, for the study, were to come up first,
to come up with a gesture, a user-defined gesture set
for text editing through a gesture elicitation study
and we also wanted to study the influence
of modern device usage on the gestures
the participants performed.
So we allowed them to use either a finger or a stylus
to perform the gestures and we also, of course,
wanted to compare our findings
with those of the earlier studies that we replicated.
And we recruited 20 participants who were all like
either undergrad or grad students at our university.
And they all used touchscreens frequently,
but they rarely used stylus on them.
And they all wrote on paper regularly.
And we used subjects designed where all the participants
were required to devise gestures for 20 tasks
and these were the 20 tasks that we used in our study.
And we labeled our tasks based on modern device usage.
So we used an Apple iPad Pro and Apple Pencil for our study
and we built a low-fidelity prototype,
which was used to present the task to the participants
and to record their gestures
and the prototype did not respond to the gestures.
And the participants were at first given a practice task
and, as mentioned before, they were allowed
to use either their finger or stylus to perform the gesture.
And because earlier studies have shown
that participants usually considered the recognition aspects
of, when devising gestures, we told them
and to not constrain that input,
we told them to assume that the implemented interface
would afford handwriting and recognition anywhere.
And they were also allowed, they were also asked
to think aloud and tell us what they,
how they expected the system to respond to their gestures.
And we began our analysis of the data collected
by grouping the gestures.
So we collected four handwrite gestures from 20 participants
and we used open-coding to group the gestures
and we grouped the gestures based
on their underlying semantics.
So, for instance, many gestures differed
in their appearances, but they were semantically similar.
Like for example, in the example shown,
for select phrase task, the first two gestures
are considered semantically equivalent, like in closing,
using either parentheses or brackets,
but the third one is considered semantically different.
And we also calculated the agreement scores
for each referent.
It is a measure that tells us
to what degree the participants agreed
on gestures for a given referent.
And you can find more details on that in our paper.
And lastly, we finalized our gesture set
by, like not only selecting the gesture
that most participants agreed on for each referent,
but we also included aliases and extensions,
which I'll talk about next.
So, aliasing involves adding synonymous gestures
for the referent,
rather than providing a single specific gesture.
And this is usually done, this practice is usually followed
to, so that the recognition rates are increased
and to provide more flexibility
and to not make it more stringent.
And we also added a couple of extensions to our gesture set.
So, if, based on the size of the tasks.
So for instance, for the select multiple lines task,
as shown in the slide,
the first gesture was the dominant gesture
that was used by most participants,
but in this, for the purposes of the task,
we had, the task consisted of text, only three lines
of text, but in practical scenarios,
you can expect the text to be much longer
and so we also added a gesture
that would consider the size of the task as well.
And this gesture also came from the collection
of gestures that we obtained from the participants,
but it wasn't the dominant one.
Now I'll talk about some of our findings.
So we first compared our gestures,
the gestures that we obtained
with those of the earlier studies,
that is Welbourn and Whitrow's study
and Morrel-Samuels and Wolf's study
and they were actually largely similar
and I'll present on examples next.
And, but there were also some differences as well.
And, all of these gestures,
like they reflected the mental model of annotating on paper.
For example, if you look at some of the gestures
for, the left side shows the gestures
that were obtained in the earlier studies
and the right side is our study.
And, so for the insert character, word and phrase tasks,
the gestures obtained were like almost identical.
And it was the same for delete, character, word
and phrase as well.
And it was the same for move word as well.
However, there were some differences too.
So we got different gestures for join,
which is equaling to delete space in our study.
And split, which was insert space in our study.
And we suspect that this was because of the phrasing
of the referent and, like I mentioned before,
we phrased our referents to be consistent
with modern device usage.
But, however, our gestures were also contained
in their aliases.
And we also had additional tasks in our study
that were not considered in the earlier studies.
And these included the select tasks,
delete long phrase and so on.
And now I'll talk about legacy bias.
So legacy bias is a term that
thatis used to illustrate the influences
of user's experiences with devices
that they've used previously.
So we wanted to study the influences of,
like modern device usage and on the gestures
that the participants came up with.
So while the paper metaphor was like, really dominant,
we also obtained a few gestures that were influenced
by modern touchscreens and the desktop paradigms.
So, for example, three participants used their stylus
to tap and hold and they expected the cursor to appear
and then they wrote the word for insert word task.
And in an another example,
one participant used the pinch and zoom gesture
to insert space between two words.
And next I'll talk about, so we allowed participants
to use either their finger or stylus,
but we found that most of them, like 90 percent of them
used stylus alone for their gestures
and the other two participants used their finger
for like a select few tasks,
but then later switched to the stylus
for the majority of the tasks.
And this is because stylus affords greater precision
and one can easily transition
between handwriting and editing
and this is in line with what is known
about stylus-based interactions in general.
And we actually found some surprising behaviors,
regarding how participants treated white space.
For example, for the insert word task,
most participants wrote the word and, like,
they found existing white space in which to write the word,
but a few others did not, like such as the second example
in the, second example shown on the slide.
So, this made us think that these behaviors probably signal
that a dynamic white space window may be,
may be an ideal method for supporting insert text entry.
So that is, when a user intends to enter text,
a dynamic window should be created
and this window should be dynamically resized
as the text is entered
and in much the same way as even using,
when typing, using the keyboard.
And another interesting behavior that we saw
was when most participants treated white space
as, white space a manipulable object,
that is they directly manipulated it.
So when they were asked to delete a blank line,
they directly crossed, they crossed it out
or, you know, like they used strike through.
But, there were also a few participants
who moved the text below the white space, upward,
to delete the blank line.
So, this was another interesting behavior that we observed.
So, and these examples suggest that white space manipulation
in the context of digital handwriting applications
can be quite complex and it needs further attention.
And, with this I end my presentation
and I'd like to thank my co-authors, Jason,
and my advisor, Ron Metoyer
and I thank you all for your time
and I'm happy to take questions.
(audience applauds)
- [Man] Thank you very much.
We have time for a couple of questions.
So if you have any, please walk
to the microphone and ask questions.
- [Man] Okay, so I will start with one.
So you presented basically just, as it were,
elicitated by your participants for text editing.
- Yeah.
- [Man] So, did you consider maybe just checking
whether the gestures elicited by the participants
were understandable in a text editing task,
like asking participants to pair from gestures
that would mean, I would like the text to be edited
like this and then giving the annotated text
to somebody else that would edit the text
and check whether it was
like the first participant was expecting?
- That would be an interesting thing to do.
We didn't do that, but yeah,
that would be an interesting thing to do.
- [Man] Was it conducted in the previous studies
that were run in the 80s on paper?
- No, so one of the studies was done on paper,
like they used paper and pencil to elicit gestures.
The other was used on one of these tablets
where, like they had a separate display for the tablet.
Yeah, so one of those tablets where they
had a separate display and the tablet was separate.
So they conducted this study on such a device.
- [Man] Okay.
- [Man] Thank you.
I didn't see your entire talk,
but what I saw was very interesting.
So, I have a question maybe slash comment,
but so, can you tell me again
what exactly you did in legacy bias?
- So.
Legacy, so for, we wanted to see what,
so how, like, modern touchscreen device usage,
like what I showed in the, like, one of the earlier slides,
so how these, so how these interactions influence,
like if they, first of all,
if they influence their gestures--
- [Man] And would that be good or bad?
- Huh?
- [Man] Would that be good or bad?
The fact that it's influenced by the legacy bias.
- Yeah, so--
- [Man] Is it good or bad?
- So, okay, so this is the thing.
Legacy bias is actually, like a debatable thing.
So some of them, there are people both for and against it.
Some say that it's necessary
to take into account people's experiences
with previous things, but some others say
that that shouldn't be done because they may not know,
like better ways to do things too.
So it's actually a debatable concept,
but we are on the side of folks who think
that it can be useful, because if they're used to something,
then they'll better remember it and, so.
- [Man] Thank you.
- Thanks.
- [Man] Okay, so if we have no further questions--