Hi, I’m Jingyi and I’m here to present on behalf of work done in collaboration with
Wil Li at Adobe Research, and my advisors Sean Follmer and Maneesh Agrawala at Stanford.
Visual content creation -- especially of 2D illustrated characters -- has been a long
standing medium for communication and storytelling, from comics, to infographics, to animations.
While professional artists have traditionally made these characters come to life in the
past, mix-and-match character creation tools, like Bitmoji, now provide a low barrier of
entry for everyone -- not just expert artists -- to create custom characters.
These tools, like Blush, shown here, allow users to quickly and easily produce characters
for their needs by compositing different clothing and accessory items together.
Although there are typically dozens of these clothes and accessories to choose from, these
tools do not support direct control over pose, proportion, or animation of the characters.
To allow end-users to easily choose and composite accessories, the artwork for these tools is
on independent layers. Body layers are layers with artwork that represent body parts, and
accessory layers have the mix-and-matchable assets.
And since each of these accessory layers are independent from the body layers, they do
not adapt to changes experienced in the body. Let's say I want to make this character's
proportions better match my own, and give them wider shoulders and a longer torso. As
you can see, the clothes do not follow this change in the body, and as a result, the character's
body shows behind the clothing.
We present an automated tool for rigging accessories by applying constraints between layers such
that the accessories adapt to deformations of the body. Here, our system deforms the
shirt so it fits to the wider shoulders.
We present an automated tool for rigging accessories by applying constraints between layers such
that the accessories adapt to deformations of the body. Here, our system deforms the
shirt so it fits to the wider shoulders.
To address this problem and allow for creating easier deformations, research has focused
on automatic rig transfer for humanoid skeletons, manipulating body parameters like height and
weight through applying a warp field, and creating single structure preserving links
for 2D illustrations. However, none of this prior work addresses our input domain of layered
characters, and our specific research question,
How can we automatically attach accessories to the body such that they deform with the
body, in turn making mix-and-match character creation tools more flexible?
Our high level approach is to construct and infer four types of constraints between layers
to generate automated clothing rigs, which I will now explain in detail.
Our system takes as input the layered artwork, grouped into body and accessory layers. Complex
accessories such as this shirt may be further decomposed into separate layers to capture
occlusion relationships.
From this input artwork, we first automate the construction of a directed acylic graph,
or DAG. The DAG tells us *which* layers should propagate changes to other layers, such as
how edits to the front arm should affect the front sleeve, but not the back sleeve, with
a directed edge going from the parent layer to the child layer.
First, users manually construct the DAG for the body layers, which means annotating the
root node, often the torso.
Next, we have an automated algorithm for attaching the accessories to the body in the DAG. For
each accessory layer, we check its overlaps with every other layer — both body and accessory.
If there is just one other overlapping layer, like the shoe and leg, we create an edge from
said layer to the accessory.
If there are multiple overlaps, like with this vest, we consider all layers within some
threshold of the maximum overlapping area. We look at the z-order of all these candidate
layers and take the one closest to the accessory — in this case, the shirt bodice.
Applying these steps for each accessory layer, here is the final DAG for this character.
Note that one limitation of our algorithm is that it only creates one edge per accessory,
while accessories decomposed into multiple layers may need more edges. Here, we also
connect the sleeves to the bodice.
The DAG tells us *which* layers should propagate changes to other ones, while our four constraint
types, which I will now explain, tell us *how* the changes should propagate. These constraints
pin the layers together at specific points to preserve different relationships, like
the skin staying hidden behind the body as the body widens.
That specific issue is largely addressed by our first constraint type, occlusion. Occlusion
relationships occur between the majority of clothing and accessory layers, such as the
arm and sleeve. They pin the boundary of the parent layer where it lies within the child
layer.
Accessory layers may also be constrained at a single point. This is the point closest
to the centroid of the child that overlaps with the parent. Layers attached at a single
point undergo a rigid body translation, so this constraint is great for accessories that
themselves should not deform, like this bowtie.
Boundaries shared between layers, common in places where fabric folds over on itself like
this shirt collar, should remain together after deformation. Thus, layers may also be
constrained along these coincident boundaries, which are defined to be within some threshold
of each other — in practice, 15 pixels.
Finally, there are cases where the entire region of overlap between two accessory layers
should remain overlapping after deformation — for example, the sleeves and the bodice
of the shirt. To preserve this overlap, we apply a constraint that pins the layers together
along the entire boundary of the overlapping region. To prevent over constraining our character,
we only place pins around overlapping regions near where the parent body parts attach to
each other — so here, where the arm meets the torso at the shoulder and nowhere else.
We also automatically infer which constraints to apply between each pair of layers. For
each edge in the DAG, we use the following heuristics:
If the edge connects a body and accessory layer together, we try to apply an occlusion
constraint, which covers the most common cases where accessories fit over body parts.
If the edge connects two accessory layers, we apply both boundary and overlap constraints
as applicable. Sometimes these operations result in no points.
In these cases, we add a constraint at a single point.
To evaluate our rigs, we demonstrate our constraints applied to a variety of mix-and-match datasets.
First is the OpenPeeps data set, which show 13 body poses and over 40 accessories across
6 types of modifications - broader shoulders, a wider belly,
an hourglass figure, shorter proportions,
taking a step, and taking a seat.
The full set of results are available in our supplemental materials.
We also demonstrate how our constraints handle custom faces. By default our inferred constraints
are at a single point, but if a user changes them to be regions of overlap, which they
may do in the UI, the facial features now themselves deform and stretch. This example
highlights how different constraints result in different aesthetic outcomes.
We also show our constraints on non-humanoid characters, like these ponies.
Finally, we show animations created with our system such as breathing -- notice how the
shirt stretches as the torso expands and lengthens with each breath — and a walk cycle -- notice
how first the pants, then the skirt, change with the legs.
The biggest limitation of our method is the lack of real world physics or secondary motion
effects — for example, as this character raises their elbow, the kimono sleeve should
drape downward rather than remaining stiff. Additionally, we currently do not synthesize
new geometries and textures so patterns on clothing may become distorted, like on this
tank top.
Our vision is to allow anyone to customize their mix-and-matchable characters beyond
just simple accessory swapping, towards more expressive and representative visual communication.
Thank you.