Hello, My name is Michihiko Ueno.
Thank you for your interest in our research
on the Continuous and Gradual Style Changes of Graphic Designs with Generative Model.
It is performed jointly with Shin’ichi Satoh at National Institute of Informatics.
Our paper contributes a novel interaction method for
exploring the graphic designs with parametrical manner.
It is difficult for novices to create a high-quality
graphic designs because they have not accumulated any ideas about layout.
The retrieval and refinement techniques for graphic designs are not
useful to a user who have not accumulated about it.
Therefore, the support and suggestion techniques that
give alternative ideas are important for such users.
However, that techniques in previous works are presented with
discrete representation such as thumbnail view.
These interfaces are not adequate for accumulating
different layout samples or mapping relationships between layouts.
Thus, we focus on the techniques to present layout design with the view that continuously change the layouts.
There are some previous works for creating graphic designs.
We categorize previous works into two categories.
One category is layout design support.
Previous works included in this category
are focused on the interface to support user creation of layout design.
DesignScape and Sketchplore are interfaces to improve
layout created by the user
with optimization methods.
DesignScape also suggests alternative ideas of the layout designs.
However, these proposed optimization methods are unable to generate
complex and flexible layouts such as those created by skilled designers.
Moreover, these works presented such layout ideas
with a discrete thumbnail view.
This presentation method is not suitable for
exploring a large number of layout designs.
Thus, we used a neural network model in this study to
generate complex layout designs and continuous layout changes.
The other category is Layout Synthesis by Deep Generative model.
Previous works included in this category are focused
on the automatic generation of the layout designs
with a deep generative model.
LayoutGAN is the parameter-based deep generative model which generates various
types of layouts such as document and Clipart.
The content-aware generative model generates graphic designs based on
the contents
such as images and keywords.
These models are helpful but they cannot be adapted directly to
continuous graphic design changes.
Our proposed method is a pixel-based generative model
to generate layout segmentation images.
Our model changes graphic designs between two graphic design
styles continuously and gradually.
To achieve this method,
we introduce a novel encoder-decoder based deep generative
model
comprising dual adversarial constraints.
The animations on the right side are the results
of our proposed model.
The left animation is the results of
the refinement segmentations
of generated layout segmentations.
On the right is the results of the graphic design rendering.
These results are automatically filled with graphic
elements based on the left refinement segmentation images.
Then, we present the proposed network architecture.
Unlike the representations of natural images, the layout
designs are discrete representations.
The regions of the elements are clearly divided.
Therefore, we introduce some techniques in latent vectors and critic networks.
First, we introduce three types of latent vectors fed into the generator.
Reconstruct variables are used to improve the reconstruction results
as used in many cases of the encoder-decoder model.
Interpolated latent variables are used to improve
the results of graphic design generation with interpolation techniques.
We also latent variables sampled from prior distribution
to improve the results with arbitrary latent vectors.
Finally, we introduce two types of critic networks.
One is the encoder constraint.
Using this constraint, the encoder is trained to map
the latent space to the Gaussian distribution
by minimizes the Kullback-Leibler divergence between prior distribution
and encoding distribution.
The other constraint regularizes the decoder by minimizing the Wasserstein distance
between data distribution and model distribution.
This constraint improves the quality of the generated
results with an arbitrary latent vector.
We conducted some experiments to evaluate our model.
We used the public dataset to validate our proposed method.
The dataset is consists of various layout semantic
segmentation XML files.
Those data are categorized into six graphic design styles
such as fashion and food.
We used three thousand nine hundred 4 images
generated with those files.
These images are composed of six types of graphic semantic segmentations
such as text and image.
Then, We encoded these graphic element types in the images
images into one-hot vectors and used them as input.
We first conducted the experiment to evaluate reconstruction.
We adopted the Wasserstein autoencoder as the baseline to validate
our techniques for the generation of graphic designs.
We set the dimensions of latent spaces to one hundred twenty-eight.
On the left side of the image are the results of
reconstruction by test dataset.
Ours are shaper and better-defined boundaries of edges than ones by baseline.
On the right side are generated samples with random latent variables.
Our proposed model outperforms the FID score
compared to the baseline.
Then, we evaluated the continuous and gradual style changes of graphic designs.
First, we filled the graphic elements with the images and texts
based on the generated layout segmentation images.
To detect the regions of generated segmentations and create
polygons for refinement of the region clearly,
we applied some computer vision techniques such as contour detection.
Then,
we got the complete graphic designs.
We conducted an experiment for a novel interaction method
that change the graphic design styles parametrically,
called style morphing.
We adopted the linear interpolation techniques to generate
layouts between two graphic designs.
We could acquire alternative layout designs by changing
the coefficient alpha.
Some of the designs generated by this method were not present in the dataset.
present in the dataset.
We evaluated the results of style morphing by a preliminary user study.
We conducted a user study to evaluate our model by forty-seven crowd workers.
They were asked about the richness of the variations
and visual appeal of generated graphic designs.
We created sixteen combinations of the
start and endpoints of graphic designs.
We extracted the generated three sample images from these combinations.
We asked workers whether they preferred a combination of A and B
for the richness of variations and visual appeal.
The images are examples of the user study presented to workers.
72% of all responses rated the
results from our proposed method as having more layout
variation than baseline.
And 74% rated the results of our proposed
method as having a better visual appeal than baseline.
We proposed a novel interaction method to explore alternative
ideas of graphic designs
Our proposed encoder-decoder model encourages to generate clear layout
segmentations by comprising dual adversarial constraints
The model enables continuous and gradual style changes
of graphic designs with parametrical manner
Some experiments demonstrated that our model generates sharper and
clear boundaries of segmentation images.
Preliminary user study demonstrated that our model have rich variations
and visual appeal.
I’d like to thank you all for your interest.