Hey.
I'm Rahul from University of Toronto,
and this work is a collaboration with my colleagues,
with Wilmot Li, Danny Kaufman and Rubaiat Habib from Adobe
and my advisor, Karan Singh from UFT.
So magical hands is about using hand gestures to alter
animations in virtual reality.
Imagine that you want to create this animation in VR.
How will you create this highly constrained
and three dimensional path of the airplane rotated in just
the right way and create all these fire effects
at the same time?
Let's consider the motion of something like the airplane
first before fire.
If you want to create such paths in current VR animation
tools such as Oculus Quill, the way
you would move and rotate and pose an object
is using widgets, which are very similar to what
you see in desktop-based tools such as Maya.
And that's not super intuitive, it's not really
using VR as much as it could.
What we would like to be able to do is something like this.
Just pick up the object with your hand
and move it in 3D space.
So we want direct control of individual objects
such as airplanes.
Let's now think about the fire effects,
or what if you want smoke behind the airplane.
Again, coming back to Quill, you don't have such high level
tools such as particle systems, which allow controlling
the physics of such phenomena.
So you'd end up tediously creating each particle
as a separate object.
What we would like to do instead,
is to treat these semantically similar and interrelated
entities in a bulk.
Wouldn't it be cool if you could just
use your hands to define what the emission looks like,
and impact all particles together.
If you wanted to change the shape of the particle,
you could just drag and drop the new shape into an emitter.
And this sort of ties into the intuitive
notion of impacting these related particles in a bulk.
And the finder in motion is controlled
by some procedural or physical rules.
And the way we would like to control these animations
is using mid-air hand gestures.
Why?
The reason we want to use gestures
is because the human hand is an extremely versatile tool
with a large number of degrees of freedom,
and hand gestures can be very expressive.
When I started this project, this
was during an internship at Adobe,
we set out to create a gestural based animation tool,
but we soon realized that it's not immediately obvious
how one would map hand gestures to animation operations.
We did not know about the human preferences for media gestures,
especially for the use of animation.
And while there is a lot of work on hand gestures,
even on media gestures, there is no work
which tries to understand user preference for animation
authoring, using gestures.
The computer graphics community has also
explored the use of gestures for a specific animation tasks,
but none had looked at animation in a general setting.
Even these specific scenarios are always
focused on character animation or on facial animation.
Even outside of the research community
there is a lot of interest in using gestures,
and we can expect this trend to continue
with upcoming gesture recognizer in hardware, such as HoloLens
2 and others as well.
But these are design explorations
and a principled exploration of gesture designed
for VR animation, was lacking.
OK.
To summarize, there was no work that we know of,
which explored the use of gestures
for physically-based animation.
Given the complex spatial temporal controls
and hierarchical relationships between objects and animation,
we decided to perform a Gesture Elicitation Study.
This idea of participatory design
by eliciting gestures directly from the users, who
was introduced by Wobbrock and colleagues in 2009.
It has been very popular in the community.
And the idea behind gesture elicitation studies
is to understand user preferences for mapping
gestures, to operations.
In our study, the participants were professional animators.
Since we wanted to build a tool for animators,
it made sense to us to involve them in the formative design
phase as well.
The study design looks something like this.
The person put on an Oculus Rift headset
which also had a leap motion sensor mounted on it.
And the leap motion sensor allows
participants to see their hands while they are in VR.
We played an animation clip in front of them.
Something like this, and asked them for step
by step instructions for creating the clip.
From this data we tried to understand
both the common operations that are
useful in VR-based animation authoring and the gestures that
should be used to effectuate those operations.
For the study we chose a total of six different animation
scenarios.
And for each of those, we showed three or four clips
in order of increasing complexity.
For example we had this tornado scene.
We first showed this clip and asked for the operations
and associated gestures.
It's a very simple rotation.
Then we moved on to the second clip
which added a path along which the tornado moves,
which is a 3D curved path as well.
And then we added this follow through effect
with the tornado bending in the beginning,
and towards the end of its path.
The six scenarios that we chose had a lot of variety
and we tried to cover different kinds
of dynamic, physical effects.
Other than a tornado, we showed a classic bouncing ball scene,
we showed a rigid body interaction
scene, a discrete particle system,
a sheet or cloth simulation with topological changes,
and a fluid simulation.
We saw some very interesting gestures
from the professional animators trying
to control a number of parameters simultaneously.
Here for example, the participant
is trying to specify the overall emission direction, how
much the smoke spreads out and the amount of randomness
or turbulence in the flow as well.
Some manipulations were more direct,
trying to physically bend an object as in the real world.
Others tried to sculpt the shape of the object that
was being controlled.
And some gestures utilized more abstract demonstration
to guide a large number of objects simultaneously.
From the study, we were able to create
a list of common operations and gestures for VR animation.
Here's a list of some of those important operations
and gestures that we observed, but the paper
has more details on this.
We also identified the operations
where the use of widgets was preferred
over gestural interactions.
We also categorized the observed interactions
into four different taxonomies and the details again,
are in the paper.
Finally we concentrated all the observations
into a set of Saxon Design guidelines
for VR based animation tools.
Let's look at some of those guidelines in more detail.
So first we notice that participants generally
did not encode information in the nuances of hand shape.
For example, if they wanted to make something
like a pinch pose, it does not matter where exactly
their other fingers are.
So slight changes in the hand pose
did not really carry meaning.
Even coarse pose recognition, as a result,
can be very useful for building a gestural interface
for animation.
Another observation was the use of gestures which
were very context dependent.
For example, abstract gestures, such as moving the hand
left or right were used to manipulate
the values of many different continuous parameters.
For particle systems this could be something like the emission
density.
How many particles are being released.
For a spring's motion, this could
be something like the frequency of vibrations and so on.
So a lot of gestures can be contextualized
to reduce the mental load on users
and to improve recognition accuracy as well.
Next, we notice that in virtual reality
participants were directly interacting with objects
and had an implicit expectation of physics,
such as elastic deformations, and other behavior as well,
such as gravity and collisions.
So it may be a very good idea to have a default physics
engine which handles these operations until the user
desires otherwise, because animators always
want to go against physics as well sometimes.
OK, so while implementing many of these guidelines
remains as future work, and we hope
this was helpful for others, we implemented some of these
to build a prototype animation tool which we also
call MagicalHands.
So there is a slight abuse of notation here.
In the Spirit app tool, we focused
on direct manipulations such as directly translating
or rotating an object, and on the controller particle
systems.
For this task we built eight gestures
which I'll go through in a bit, one by one.
We also built a UI panel with non-gestural widgets
for timeline control.
And as I mentioned during the design guidelines,
we can build a gestural system using Coarse Pose Recognition
And to demonstrate this, we implemented the system
using the Oculus Touch controllers
as the posed recognition technology.
And the controller only detects if your fingers
are touching certain buttons, or pressing those.
So this is very coarse information,
we use this to infer four different hand poses.
Let's look at the direct manipulation gestures first.
We built a one-handed translation gesture,
where you just take and move an object.
A bimanual gesture with the two hands
moving symmetrically to scale an object uniformly,
and a rotation gesture where one hand formed an axis,
and the other controlled the amount of rotation
that the object goes through.
Let me employ this airplane example again
to see how this works.
You can see the use of the translation gesture
to define the path of the airplane in 3D, like so.
Then the user utilized the rotation gesture
to keyframe its orientation.
And one thing you can notice here
is that we can use performance-based as well
as keyframe-based animation in a single coherent interface
over here.
And finally, the user was able to produce this result.
OK, let's look at particle emitters now.
For emitters, we define a unified gesture
which controlled the emission curve,
the spread of the emission, as well as the speed.
This follows, again, from our guidelines of using physics.
The user defines the properties of the particle system
as a bulk, and individual particles simply
follow a physical or a pyramid rise trajectory.
We also define the tap to select gesture,
which implements the design guideline
for contextual gestures.
In our prototype we only had contextual gestures
with particle systems, but one can imagine
using contextual gestures in more complex settings,
as well as to reduce the ambiguities
in gesture recognition.
So when you do the tap gesture it
enables certain contextual gestures of particle systems,
and the ones we implemented are these three.
First the noise gesture adds randomness
to the particle motion.
If you do this.
The spiral motion gesture makes the particle swirl
about the emission axis.
And finally, users could just drag and drop an object
into an emitter to make the dropped object the new particle
shape for this system.
Let's look at this example to see
how these gestures are used.
And this is a very relatable example.
Here's a graduate student working on a deadline.
He's covered in a storm of papers.
I think it's relatable to others as well.
So the first thing the animator does,
he positions a special emitter object in the scene
and uses a water drop as the particle
to make it rain over the poor student.
Then he changes in mine, and uses a paper airplane instead,
which I think makes more sense in this scene.
To add some more drama he changes
the shape of the emission cone to make it more like a funnel.
And then he uses a noise gesture to add some randomness.
And finally, a spiral motion gesture
to create the intended tornado effect.
And there you have it.
The intended result is the graduate student
covered in papers.
All right.
So we've seen now what the prototype looks like
and how we utilize our design guidelines to build it.
We tested the system ourselves, and with a few users
in informal free form scenarios.
Let's look at some of the resulting animations
built using MagicalHands.
We created this waterfall scene which
shows a use case in education.
And you can see that our directed particle
systems really form the heart of this animation.
You can see the clouds and the rainfall.
And if you squint really hard, you
can probably see the whale's water spout on the top right.
This is a simple scene created by one of our participants.
I think what's happening here is that this devil-like ball
caused the airplane to crash, and now it's jumping with joy.
But notice the use of the particle emitter
to create the fire, which really accentuates the scene.
Here we have used three co-located particle emitters
for the bubble emission.
The dark spot was created using a direct manipulation,
and the rotation was keyframe.
And note that the control of particle systems
is clearly very important to show the intended effect here.
And this is a more complex scene created by a participant.
The wizard tells a scary story which
scares the little ball, causing it to hide behind its father.
And then there's a ghost that appears
who was hiding in the tree.
Notice the subtle randomness though,
in the fire, campfire and the zigzag shape
of the emission, both of which were deliberately
set by the user.
And by the way, all of these animations
took between 5 and 30 minutes, which includes time
for arranging the scene, experimentation,
and authoring the actual motion.
However, the geometry is important
and we do not have tools for creating geometry.
And we're pretty happy with our product tool and on the results
that we were able to generate with it,
but there's lots of interesting avenues for future work.
One potential UI exploration, which
came up a lot during our study, was
the notion of interacting directly with the gestures.
Imagine that you've used a gesture to create a path
and now you use that gesture itself as an object
which you can directly manipulate.
Another design guideline which we didn't get to explore yet,
was the use of the same gesture for coarse to fine editing.
Imagine using this gesture for the emitter
to define the path, the speed, and the spread
to begin with, but later in the work flow when the user is
making final edits it can be contextualized to only edit
the emission spread.
Sorry.
The last one.
And another very cool technical and UI challenge
is incorporating gestures with defined direct motions, as well
as physical forces.
So how do you tell a user what's possible or not?
How do you enable downstream editing
when combining both of these together?
At a higher level, I imagine using media gestures
for more creative tasks.
Personally I'm most interested in visual tasks such as 3D
modeling and sculpting, but maybe non-visual tasks
such as the creation and editing of spatial audio, or music
could benefit from expressive hand gestures as well.
Think about something like playing a theremin.
In the end, I would like to acknowledge
the contributions of Seth Walker from Adobe,
our participants and reviewers, and the artists who
provided all the nice models, which we used
for creating the animations.
And that's all.
The code for the prototype is now available
and an executable is available as well.
So please download, give it a try, and see if you like it.
We're open to feedback.
And I'll end with that.
Thanks a lot for your attention.
[APPLAUSE]
Thanks Rahul.
Do we have time for a question?
Hi.
Thank you for your talk.
So did you measure the discoverability
of the gestures?
So the [INAUDIBLE] of the authoring process,
does it contain the learning process?
And how do you introduce the gestures to the users?
Thank you.
We did not explicitly measure the discoverability,
but something that we observed, which
I talked about a little bit, is this notion
of contextual gesture, which I think
can really help with the discovery, with-- sorry.
Well, that's for recognition.
No we did not test for the discoverability.
Yeah.
OK, let's thank the speaker again.
[APPLAUSE]