- [Brad Myers] Thank you very much.
It's a great pleasure to be here
and to give this talk.
Steven apologizes that he could not be here
to give it in person.
He and his wife are expecting to grow the CHI community
by one very soon.
And so he felt like he needed to stay in Ann Arbor.
So everybody here has used multi-touch applications,
on tablets and smart phones,
and even on your watch.
They're usually very easy to use,
and pretty natural.
But the problem is that they're very hard to implement.
And one of the reasons this is,
is because of all of the different pieces
of information that you have to control.
So most multi-touch frameworks,
that you might program with,
have some fairly straightforward event types;
touch start, touch move, and touch end.
So these are pretty straightforward,
they seem pretty simple to use.
Each of these has a variety of properties
that you can use in your program,
such as the position, the ID, the time stamp,
and for many devices, the force.
The problem is that when you try to use these
in a gesture that fires,
when you have two fingers or multiple fingers,
and you want to have movements,
then it gets to be a lot more complicated.
So let's see what the requirements are
for the different events.
So rather than writing code,
I'm just going to list the specific requirements.
So first, for the touch start listener,
in order to know whether the user should start the gesture,
first we're going to have to count the number of fingers.
And that count has to be a global variable,
so I'll put it in this gray box to show that it has to be
outside of the gesture handlers.
And, of course, this global variable
is going to have to be incremented,
and then read from the start handler.
You'd also want to, of course, clear it
from the end handler.
And, you probably are going to want to have a time-out
to make sure that the two fingers touch the screen
at about the same time.
And that also has to be read in multiple places.
Also we might want to know that the two fingers
go down close to each other.
And then you have to measure the location.
And that, of course, would have to be updated
based on if the first finger moves
before the second finger goes down.
And, also that also has to be dealt with on release.
So, we have all of this code already
just to make sure that you have two fingers down.
But now we have to deal with moving them,
which has a bunch of other requirements and code
and call-backs that you'd have to do.
So, all these also can happen in almost any order.
And we're dealing with event handlers
and global variables.
So all sorts of multi-processing kind of coordination
has to come in as well.
So what could we do to try
and simplify this level of complication?
And to try to have more understandable primitives
that you could program out of.
And so what we are doing is to try and understand
how we could make this better for programmers.
And the first thing we did is a pilot study
using this technique we call
Natural Program Elicitation,
which is where we try to understand how programmers
would naturally want to express these ideas.
And typically they write them down on paper
with a pencil, not necessarily even using a computer.
And they've, we ask them to invent
any primitives they want.
And they can either use these in code and text,
or they could even draw pictures.
So when we ask them to write code for the press,
remember we had a lot of requirements
that they would have needed otherwise.
But what people tend to write
is just a fairly simple press.
And they would have an event that said
two fingers down, and that would be the event they have.
And then when we ask them about
how they would naturally express movement,
instead of writing a bunch of code
they would typically draw a picture,
like this vertical line,
to show what they had in mind
in terms of the reference for the event.
So, we're inspired by these natural ways of expressing
things to provide a toolkit with two different
fundamental operations in it.
One is touch groups,
that you can use to express
a group of fingers doing an operation.
And then it can fire a single event
for your application.
For example, when two fingers go down.
And then we added cross-events,
which is a set of events you can use
to describe the movement of the touch groups.
And they provide an easy way to identify
when the touch event goes past a particular position.
In order to support the kinds of behaviors
that people want to express,
we arranged for the crossing events to be an arb itory path
so you can define any path at all with SVG.
But most of the time people just use lines and circles,
cause those are the most common and simplest thing to do.
Another important aspect is that the cross event parameters,
like where they are physically on the screen,
can be computed with constraints.
So a constraint is a equation
that the system maintains for you
and this allows the paths to be dynamic.
They can depend on where the event starts and so forth.
So here's a simple example,
we can say that the X position
of the crossing event,
would be based on wherever the touch goes down.
That one's not particularly useful,
much more common would be to say
the X position depends on where gesture starts.
Right?
And then so you can tell
whenever the fingers move more
than 10 pixels away from where they started.
And these paths can be any shape,
for example, if you wanted to make sure that somebody
presses down and does not move,
then you could draw a small circle around
the original location and say
cancel the event if it ever moves out of
this little circle.
So how would you use those events
and crossing paths in order to program our little example?
So first we create a touch group,
and we'll call it T,
and we can specify that it has to be
two fingers down at the same time.
Everything else we leave as a default values.
And then we define a path P,
that's again to the right of
where the gesture starts,
and let's call it 20 pixels away.
And in terms of the vertical distance
it would have to be, say,
30 pixels above and below where I started pressing.
And that's all you have to do, really.
So now you can say
when this particular thing happens,
when the touch group crosses this line,
then whatever code we wanted to have happen
in that particular situation
would then occur.
So that's kind of very simple,
that's a lot of what you need,
but once, as soon as you have things like
crossing and multiple presses,
you immediately have these two conflicts
that multiple groups might fire at the same time.
And so we have a whole system of priorities
and different flags that you can use
in order to fairly straightforwardly
prioritize between the different kinds of events.
So in order to understand
to what extent this system was usable by programmers,
which, I'm sure, many of you know
is a really hard thing to identify,
we ran several different evaluations.
The first one we did was a readability study
where we wanted to make sure that people could understand
our crossing event kind of code.
So what we did to do that test
is we showed people the two different kinds of code,
in this case the touch event code on the right,
and conventional java script on the left.
And then we asked them to evaluate them.
This study lasted about an hour
for each of 18 participants.
They all had prior programming experience
and user interface experience.
And they had basically 20 questions.
Ten with crossing events,
and ten with conventional java.
Java script, I mean.
And at the bottom is basically the test,
they're suppose to look at the code
and pick which of the gestures at the bottom
this code implements.
Naturally we controlled for orders,
so half of the group got the touch event first,
and half of the group got the java script first.
And the kinds of gestures that we tested
were standard gestures that you would imagine from
any of your common apps,
as well as some custom gestures,
discrete and continuous gestures,
also static and dynamic gestures
where the value moves like scrolling or zooming.
Where you want to end up knowing the time
basis of the gestures.
And then, of course, one finger
and multi-finger gestures as well.
So we didn't see any difference,
in the different kind of gestures
across the performance,
But we did see a statistically
significant difference in both the accuracy
and the time for reading the code.
And luckily for us,
the controlled group was worse,
our experimental group did, in fact, do better.
We interviewed the participants after the study
and they said nice things.
The standard touch events
were easier to understand on paper.
So just conventional down, up moves sounds easy.
But it turns out to be pretty difficult
when you actually see it in the code.
Another participant said the standard touch events
load a lot of information into
those three functions,
so it can often be difficult to read
and understand quickly.
Event handlers like that are just generally
kind of a mess to read.
So it's also important that these,
this code can be written,
and so we did another user study
where we asked people to write some code,
We had 10 different participants in this study.
Again, it lasted about an hour.
And we had three different conditions
that we asked everybody to write
their code in.
Obviously we trained them on all these three
and controlled the order.
In the control condition
they just wrote code using conventional
touch events in java script.
In the first experimental condition
we used the touch events embedded in java script.
The touch groups embedded in java script,
as I was showing before,
and this time we added a third experimental condition.
Which was having a graphical user interface
to represent these gestures
in kind of a declarative style.
And this integrated with Steven Ones' thesis system,
which was called "Interstate,"
which was a graphical way doing transition diagrams,
So in this case they could define
the same events as they did before.
But this time there was menus
that would pop-up the different options,
and so forth.
So, again, we asked the participants
to implement three different gestures
taken out of a large set of gestures
that were standard.
And more custom kind of gestures.
We randomized the assignment of gestures to
conditions and the order in which they were done.
I should have mentioned before,
by the way,
that in the readings experiment,
we had an external person write the code.
So it wasn't any of the experimenter
who wrote the original code.
And similarly here,
we had an external person grade the implementations
to try and avoid bias.
And we tested their code
in a variety of pre-scripted situations.
So unfortunately we didn't find any
statistical significant difference
either in time or in accuracy
for the write-ability study,
although the trend is in the right direction.
And certainly we can say,
it certainly wasn't any worse,
Even though these people were highly experienced
in writing conventional touch code in java script.
In the interview, the participants were very positive.
The ability to specify the path
to say exactly where you want the event
to actually cross was helpful.
Instead of trying to calculate it yourself.
Just to be able to make a circle,
to make an event handler for crossing that line.
Instead of trying to calculate where those points were
and where they are now,
It's nice that the circle can update its position
based on the object.
So beyond readability and write-ability,
we'd also like for this mechanism to be general purpose.
It can be used in lots of different situations.
So we went to the prior CHI literature,
collected a whole bunch of different gestures
that people had reported in previous papers,
that CHI enlists and so forth.
And we implemented those in touch groups,
and the details are in the paper.
So, in conclusion, we introduced
two new primitives for multi-touch programming,
touch groups,
which describe multiple touches
that happen in synchrony.
And crossing events,
which summarize the movement of those touches.
We conducted a readability evaluation,
where we found that participants in fact,
were able to better understand
than imperative code written with touch groups
and cross events.
And then we conducted the write ability evaluation,
where we found promising results .
But because of improvements with time and correctness,
those were not statistically significant.
We can only really say that participants
performance was on par with standard touch events.
Thank you.
(applause)
- [Moderator] Do we have any questions for the Speaker?
We put four microphones in this room.
One here,
one over there,
and two at the bottom of the tier seating.
- And I have to say that I will
try to answer any easy questions,
but if you have hard questions,
Steven did an awful lot of work on this on his own,
after he graduated.
So you'll have to send him an email.
- Hopefully an easy question, Brad.
Jeremy Corinstock from Magil University.
You're the tools man.
You've been at this for so many years.
And I'm fairly new to this space,
but I just can't imagine that multi-touch primitives
are not already a staple built into OS'
of all of these underlying systems.
Why is it that such a study is still relevant
in today's age?
- Yeah, so if you look at the Android or IOS,
they basically have touch start, touch end, touch move.
Which is fine for single finger programming,
but really doesn't help pretty much with
multi-finger programming.
And they have a lot of the standard operations built in.
So two finger zoom or two finger scroll,
you just get a scroll event.
You don't actually have to handle that yourself.
But a lot of standard apps that we use all the time,
including Google Maps,
have custom multi-finger behaviors,
and it's still the case that you have to
program those by hand from the level touch events.
- [Jeremy] Shouldn't this reflect poorly on our community?
- (laughs) I'll certainly agree with you on that.
- [Moderator] Is that our questions?
Don't be shy.
OK, so I have one.
Could the cross event framework be used to capture
more complex trajectories?
So stroke gestures, for example.
- Yeah, so, obviously it's not appropriate to use this
for things like character recognition.
You could do some simple crossing,
like if you wanted a C gesture or a circle gesture.
That could be done.
But, we certainly didn't intend for this to be like,
a recognizer for hand writing or
some really complex gestures.
But through the survey of almost all apps,
on phones and tablets,
and almost all of the research papers,
we can really cover a substantial number
of the gestures that people really use.
- [Moderator] Any more?
One last one then.
Do yourself or Steve have plans to release
of these tools so others can use this themselves?
- So this particular implementation is built into
Steven's "Interstate" system,
which is available for downloading.
But I don't think he has any particular plans
to release it separately.
But there are enough details in the paper
that you could reproduce it fairly straightforwardly.
- [Moderator] Let's thank our Speaker again.
(applause)