- Thank you for the introduction.
So, I'm going to present our project about
how you can leverage crowdsourcing
to create interactive prototypes
at early stage of UI design.
This is a collaborative work with my colleagues;
Yujin, Isabelle, Yiwei, Stephanie,
and we are all from Cro-Ma Lab,
which is directed by Walter Lasecki.
Creating a sketch prototype is an essential step in user
interface design, and it helps designers quickly explore
different ideas and use it to communicate
with their colleagues because it is easier to get
a sense on how the UI will look like.
However, it is not clear from a sketch to know what
is going to happen when a user interacts with it,
'cause it it not interactive, rather static.
Indeed, a previous study reported that designers express
difficulties in prototyping UI behaviors.
Typically, what designers do is to describe the behaviors
in text, and annotate on the sketch.
However, if you think of the actual behaviors on
the final products, like you see on the bottom,
it would have been difficult to get the same feel
from the textual description.
For example: When Steve Jobs demonstrated "Swipe to Unlock"
ten years ago, people were applauding and screaming.
However, you would not have the same excitement
if it reads "Swipe to Unlock" from the sketch.
Our goal is to develop a tool for prototyping
interactive behaviors that is agile for rapid iteration,
easy to use for non-experts, and expressive enough to
support general UI application.
We have a solution for the appearance of a sketch,
which is pencil and paper, which provides expressive
and natural ways to create a rough sketch for UI.
So our question is: "What is something equivalent
to pencil and paper for prototyping interactive behaviors?"
There are potential solution to address these challenges.
For example: Most of UI Design software these days
provides a library of animation that you can choose from.
However, oftentimes the library is focused on certain types
of apps such as mobile apps or website transitions,
and it requires a high fidelity prototype, which makes it
difficult to explore ideas at an early stage.
Another potential solution to this problem is to use
the tools that are developed in
the context of creating animation.
This case sketch, for example, provides an easy to use
environment where a user can create an animation by
demonstrating behaviors and later editing it
on a timeline editor like film editing software.
We drew lots of ideas from case sketch, in fact.
However, AFSci case sketch produces one linear animation
and it does not scale well for one UI
that contains multiple interactive behaviors.
Lastly, the tools developed for interactive illustration
can be useful to quickly explore ideas.
It provides rich expressivity and interactivity,
but this is still kind of an and user programming,
which requires competitional thinking.
So, all of these approaches are valid in its context,
and, however, each of them have some limitation
that makes it difficult to use for
UI prototyping at the early stage.
One of the effective methods is to create a paper prototype.
In the manual they'll demonstrate the behaviors.
As you can see, it can be very labor intensive and
time consuming to create the prototype
and perform the demonstration.
Another problem is demonstration is ephemeral,
so you would have to videotape the demonstration
to archive it, and our artifact, which is paper prototype,
does not know anything about the behavior.
Lastly, there are certain things that cannot be done
manually, 'cause simply humans cannot do it
because of the limits of our physical motions
or the materiality of the paper prototype.
As an alternative, Apparition address the problems
of manual demonstration by collective crowd workers
collaborating on a real time shared canvas.
In the user study, the requester asked the workers to
demonstrate simple behaviors, as you can see in the video,
like moving the character to where the requester clicks.
Still, the system shares the similar
shortcomings of manual demonstration.
To address this problem, we introduce SketchExpress,
a web-based sketching system that helps designers
prototype complex interactive behaviors.
It takes the idea of crowdsourcing from the
previous work of Apparition, and it is,
in fact, built on top of Apparition.
So, in SketchExpress, a requester or a designer
verbally describes the interactive behaviors they want,
and the crowd workers that are connected to the system
listen to the verbal description,
and create interactive behaviors in real time.
We provide an easy and expressive method called
"Demonstrate and Remix" to make the static sketches dynamic.
The method allows non-expert crowd workers
to create animation in a few minutes.
At the end of the day, the designer will have a sketch
that includes multiple interactive
behaviors that can be triggered
individually or simultaneously by anyone.
Our eventual goal is to have an intelligent system
so the designers can hand off their work
of refining UI to crowd workers.
As our requester's interaction with the system is natural,
non-expert designers will be able to
use the system with minimal training.
In this paper, we focused on the method that
we've provided to the crowd workers.
I'll explain the technical details of the
"Demonstrate and Remix" method from now on.
In SketchExpress one can create an
interactive behavior by demonstrating it.
So this is the interface that crowd workers use.
A requester will see the same thing
in real time on the canvas.
Once the requester will speak to the system
and verbally describe the desired behavior.
Let's say a requester wants a crowd worker to create
a behavior of Super Mario moving forward and
jumping forward, and then the crowd worker listens to
the verbal description and start recording,
name the animation, and demonstrate what is supposed
to happen in the canvas for that particular behavior.
Once created, one can replay the animation
and any changes made in the canvas
during the recording phase will be reproduced
as a behavior in real time as it was demonstrated.
You can see that on the right side, as a worker
demonstrates, the system knows what kind of operation
that the worker makes on each element, for how long.
Once their recording is complete, one can reset
the animations so it can quickly restore
the initial state of the animated elements,
so that the behavior can be demonstrated again.
Because requesters' interaction with
the system is natural, they can freely, uh-
it's fluid, suppose the requester asks
the crowd worker to come up with a
different kind of jump behavior,
and the crowd worker can be creative
to satisfy the subtle requests.
So what happens if I replay the behavior one more time?
SketchExpress does not replicate the demonstration
as a series of snapshots of the element.
Instead, it's going to repeat the behavior
based on the current state of the element.
This is because recording algorithm computes the
difference between each event
during the recording phase.
And, they will replay based on the calculated difference,
which we'll call Delta here,
and this will work for any refined transformation
such as translate, rotate and scale.
Recording a replay by Delta will expand
the expressivity of animation in SketchExpress.
Here is an example,
let's say I recorded an hour hand of a clock
moving from one o' clock to 3 o' clock,
which is basically rotation by 60 degrees.
If the behavior is replayed again,
the hour hand will move from three to five,
five to seven and so forth.
Now the behavior is not moving from one to three.
Instead it is two hours of elapsed time.
Another benefit of replaying by Delta happens
when a user triggers multiple behaviors simultaneously.
This allows complex interactive behaviors
and needs concurrent action for example.
If I want to express a rolling stone,
there's no way for a crowd worker
to manually demonstrate the behavior
because rotation and translation of an element
occurs in two exclusive modes.
However, separately recorded animation
of a rotation and translation can be replayed together
to express the rolling stone animation.
Most importantly, SketchExpress allows crowd workers
to remix the demonstrated behaviors afterwards.
Suppose the crowd worker is asked to create
a behavior of a mushroom dying,
like in the top right picture,
first the worker will simply record the behavior
based on what is supposed to happen.
However, the demonstrated behavior is
far from the example on the top right.
It is because the recorded behavior
happens in real time as it was demonstrated.
For example it takes too much time to rotate the mushroom
and it does not immediately translate to the next motion.
In this case, a crowd worker can use the remix function
to adjust the timing of the behavior.
In the remix function, a crowd worker can choose to
change the duration of each operation individually,
to replay an animation instantly,
or even skip the operation if not necessary.
For example, in this case,
the rotation operation should happen instantly,
and we don't want any delay between any two operations.
And the worker can adjust the time
of the move operation as well, to make it more dynamic.
As the behaviors get remixed, you can see
that the worker accomplishes the seamless behavior,
which is close to the example on the top right.
Lastly, the created behaviors of an object
in SketchExpress, so that you can reuse it
for other visual elements in the sketch.
For example, let's say the turtle has it's own behavior of
hiding in a shell when Mario jumps on it.
However, if the turtle moves in the same way
that the mushroom moves, or die in the same way
as the mushroom dies, we can reuse the behaviors.
A crowdworker can simply create a duplicate of
the interactive behaviors and apply it to another element.
This is reusing the behaviors in action.
Suppose the behavior of the mushroom dying
can be applied to the turtle.
A crowd worker can clone the behavior and rename it,
and then one can apply the behavior
to a different element in the remix function.
In this case, the turtle.
The crowd worker simply needs to click the element
and apply the behavior to it.
There are a set of functions in the the program agenda
that we didn't explain in detail for the sake of time
such as looping, concatenating, inserting.
Please refer to the paper for more details.
So, this is an example
of how far we can go with SketchExpress.
This is done by one of our undergrad students
demonstrating the game Flappy Bird.
As you can see, the game character, the background,
and the game over screen itself is animated independently.
Imagine how long it would take to demonstrate
these kind of behaviors by programming
or how many people you would need
to demonstrate this with paper cutouts.
You may be focusing on the left side,
but the more interesting things
are happening on the right side.
By pulling the levers of right animation, right time,
one person can accomplish to demonstrate
how the game works effectively.
Automating the process of manual demonstration
is another benefit of using SketchExpress
and all of this could be done
in tens of minutes, solely by one person
Another thing to note is, this is non-linear.
So based on the order and the timing
of triggering behaviors, you can demonstrate
a completely different scenario.
Like, where Flappy Bird reaches the ceiling,
or it dies because it goes out of the screen.
To validate the effects of SketchExpress,
we conducted a user study with 18 crowd workers,
recruited from Mturk.
We had three different conditions.
The first condition, or C1, is the baseline condition
of Apparition, where crowd workers need to
manually demonstrate a described behavior.
The second condition, or C2,
is the one that has functions
of demonstrate and replay, but not remix function
We added C2 for a condition to
identify the effect of the remix function.
The third condition, or C3, is the SketchExpress condition.
One of the author acted as a requester
by reading pre-written script of describing the behavior.
We had five different UI scenarios
from games to traditional GUI to the list.
We had in total nine different behaviors
that we asked to demonstrate or create.
We told workers to demonstrate the behaviors twice
and allowed them replay the created animation
in condition two and condition three,
which had replay functions.
We analyze the screen recording
and evaluate it with a set of
binary rubrics per demonstrated behavior.
For more detail on the analysis, please refer to the paper.
So we confirmed that the effect of both
replay function and remix function.
The figure represents the precision and recall
of these, the three conditions.
C1 baseline, C2, the one without remix,
C3, the SketchExpress.
Increase in both precision and recall
compare to the baseline condition
was statistically significant.
In terms of precision, being able to skip operation
helped accomplish 99.2 percent in precision
because workers can simply skip
any extra operation made during the recording.
Also, we could confirm the effect of remix function
yielding the increase of 27.5 percent in recall.
The remix function significantly improved
the quality of the demonstrated behavior.
Again, we asked crowd workers
to demonstrate the behavior two times.
This figure shows that completion time of the first trial
and the second trial for three conditions.
Apparently, creating behaviors in SketchExpress
comes at the price of needing
more time to refine the behaviors.
Still, workers were able to create
one interactive behavior in 174.5 seconds on average.
Which is less than three minutes.
The real benefit comes from the repeated use of behaviors.
For the second trial on condition two and three,
workers just need to press the right replay button.
So, once the interactive behavior is created,
now it is a matter of human competition
to find out which button to press.
Crowd workers really enjoyed the task,
sometimes more than we hoped.
One observation was that they spent
too much time remixing and retrying the animation
while it was already good enough based on our rubrics.
For example, one worker spent more than 13 minutes
to make the behavior look perfect.
In practice, a requester will give
immediate feedback to stop them refining.
Lastly, we didn't do any form of survey
with crowd workers, but they voluntarily sent us
very positive feedback after the sessions.
In summary, the last study reports that the
crowd workers can prototype on interactive behaviors
within three minutes on average
with 27.3 percent gain in recall
compared to the baseline conditions.
Our contribution will be the method of demonstrating
the remix and replay method which can be used
in any kind of relevant software
and the system allows crowd workers
to create interactive behaviors.
One take away of the study is that
we can mitigate the challenge of
real-time collaboration ironically by bringing some
asynchronous interaction technique
such as recording, replaying or remixing
which relieves the real-time pressure.
Our bigger goal is to create an intelligent assistant
for GUI sketching tools Apparition and SketchExpress.
Let us explore the basic functionality
powered by crowd sourcing
and we are excited to take this further.
Immediately, we would like to understand the challenges
that the requester would have to verbally communicate
and to collaborate with crowd workers in real time
through qualitative user study.
In addition, we'd like to take advantage
of the crowd's creation and demonstration
to automate the process of creating a GUI sketch
and computation will regenerate the sketch based on
the requester's description and minimal visual input.
Thank you for your attention,
last but not least I'm on the job market this year.
So, if you need a technical AFSci person,
a researcher, or teacher,
or all of the above, just let me know.
I'll be happy to take any questions.
(crowd applause)
- [Steve] Steve Finer, Columbia University.
This is really neat stuff,
and you're positioning it for prototyping
and early sketching, right?
But, in this kind of, like, eight-bit graphics look.
Which a lot of people really like,
even for finished products.
It looks like it could be an interesting way to make
games that are continually maturing and evolving,
maybe courtesy of having crowd interaction
so the game actually changes in interesting ways
from one day to another and it's not just
an early prototype, it's actually
the real thing that people are playing.
Have you thought about that?
- I completely agree with your thought.
Actually, the stuff that you see here,
you see the three arrow buttons down on the bottom?
I actually just fiddled with a crowd worker
and then I was clicking one of the buttons
and then I asked the crowd workers to
replay the behavior based on what I clicked
and they were kind of playing the game
and there was like, eagles trying to
catch the chicken, and wandering around,
and then I asked the crowd worker to
come up with something creative he or she want to have
and then I thought, this person will come up with
something about the game but, like,
this person created this animation,
that sun rise from the horizon
so it was very rewarding, but I like the fact
that the crowd worker's take was
something that I had never thought of
and then it actually will facilitate creativity,
in some sense, for the end users.
Does that answer your question?
- [Steve] Yes, thank you.
- Thanks.
- And thank you very much.