welcome to the talk of her garment
capture and synthesis for real-time fur
to try on in this talk we discuss our
system for live virtual garment try on
and hi thank you for coming i'm toby
chung from the university of tokyo i'm a
first year phd student under the
supervision of tokyo igarashi for you to
try on which allows you to try different
garments virtually it's important for
two main reasons
first we're shopping for fashion items
online more and more often these day
and when we do that we don't really know
how piece of garment would look on our
body before we actually get the item
days later
and secondly refer to conference being
more and more popular than ever many
people are looking for a solution in
which they can attend formal meetings in
the comfort of their pajamas
existing were the attempts to address
the problem these methods replace the
garment of a user in an image given
another different garment reference
however there are a couple fundamental
limitations that prevent this method
from being practical for live try on
namely limited data set and the lack of
size reference
first these methods are often trained
on professional model images which only
include a very small range of all
possible motion during virtual try-on
which does not cover the rich motion
that people actually do when they
actually try to test a piece of garment
secondly and arguably more importantly
these methods fail to capture the size
of the common train using only a single
image of garment without the size
information of either the garment or the
fashion model most methods do not have
the concept of size
and cannot generate accurate size of the
garment on the user's body
this is especially problematic when it
comes to virtual try-on for online
shopping where the shopper is offer
multiple size options to and has to
decide which one they prefer
based on the limitation that we observe
we introduce a virtual trial on system
that allows real-time interactive
garment to try on
and here are some of the results
here is the overview of our method
our method can be broken down into two
main parts
try on which is where we generate the
result of the user in a different
garment in real time and capturing where
we prepare the data for the try on part
our key contributions are first we
design a mannequin robot that allows us
to capture garment extensively
and second we designed a proxy
measurement garment that we provide a
size reference for the user
we first design and build an actuated
mannequin with nine degree of freedom
is made of 3d printed outer shell and
sheet metal internally
this mannequin can rotate and move its
arms to mimic different pose that a
person would do during actual try on
additionally the mannequin can change
its shape into different body
configuration
with the mannequin we can capture
garment deformation under different body
pose and shape
for each piece of garment we rotate the
mannequin and change the opposition and
body shape and capture 80 000 images for
the garment this process takes
approximately two hours and this process
is mostly automatic except at the
beginning where we have to put the
garment onto the robot
we also design what we call a
measurement garment a measuring garment
is what a user has to wear to use our
system
it's a color-coded t-shirt which marks
different parts of the user's body
we designed the measurement garment to
be something that we can easily extract
from a video frame and it's something
that is easy to put on and it should
retain the body pose and size
information
we test a couple design of the
measurement garment before arriving at
our current version of it
the measurement garment that we design
is inspired by segmentation
virtualization and is designed to be
bright such that it's easy to be seen
even under darker indoor environments
now that we have introduced all the core
building blocks of our system we can
talk about how we prepare the data
required to produce
the virtual result
first thing we need is to capture the
measurement garment this happens only
once and we can reuse the same data for
any new garment that we want to provide
try on for
to generate a virtual garment for a new
piece of garment
we need to capture the garment using
dominican robot
and finally we use the data featuring an
image to image translation network which
takes approximately 2 days
we capture paired data for the
measurement and target garments which
are the data under the same body pose
and shape
we then encode the measurement comma to
our segmentation map using k-means
then we adopt network architecture from
page to pix hd featuring a network to
map the segmentation map to the target
garment
conceptually we basically learn a
mapping function that converts the
segmentation map
of the measurement garment to the target
government rgb values
and retrain it using both adversarial
knot loss and feature matching loss as
described in the pix2pix hd paper
and this concludes the preparation
process of our system
and that brings us to the try on part of
our system
where the user wears measurement garment
stands in front of the camera and we
generate a virtual charm result for them
we do so by first separating the
measurement garment from the frame
using a simatic segmentation network
after removing the body and the
background we apply k-means to the
measurement government to obtain a
segmentation map we then pass it through
the image to image translation network
that we trained earlier
we can load different network weights at
runtime to dynamically change the
virtual garment in real time
and finally
we can recompose everything including
the background and the user's body and
the comment together to generate the
final result
here we show a couple example results
generated using our system
additionally we also show result
generated using an existing method acgpn
which uses only static product image as
the common input
we evaluate our method using a couple
studies first we conduct a user study
where participant can try our system and
provide feedback
second we also ask the participant to
identify the size of the garment that
they prefer using only our system
and finally we test whether our system
is robust under different lighting
conditions
for the user study we recruited a
participant and they were able to test
10 virtual government in total five of
the government of different designs and
five of them of the same but they are of
different size
the user study feedback is generally
positive with the participant finding
the system very useful and provide
realistic results
we also asked the participant to test a
virtual garment of five different sites
and tells us what size they prefer using
only our system
after that they have to try the garment
the physical one and tell us which one
they prefer after they actually put it
on
with a proper reference of size using
the measurement garment as a proxy our
method can produce accurate government
size relative to the user's body while
the baseline method ac gpn cannot
seven out of the eight participants
correctly identified the size they
prefer using only our system
our method can also be applied to video
conference application where only a
portion of the upper body is visible
in summary we introduce a real-time
interactive virtual try-on system using
a data capturing mannequin robot and a
specially designed measurement garment
you can learn more about our project and
updating some of the data from our
project page thank you