Hi, my name is Enhao and today I will present our interactive method for visual exploration
and examination of Generative Adversarial Networks, or GANs for short.
Recently, GANs have shown promise as creativity support tools and have been used to create all kinds of interesting images.
Here I just want to quickly introduce GANs before we move on.
Training a GAN involves two networks, a generator and a discriminator.
The generator takes in a latent vector z and outputs the corresponding image.
During training, the generator tries to fool the discriminator, while the discriminator
is trained to distinguish between generated images and real images until it can no longer tell the difference.
In this paper, we are interested in assessing the quality of generated outputs.
For example, we want to know if it can generate photo-realistic images.
However, evaluating GANs is challenging, and the most common way to evaluate GAN models
is still using tedious visual examination of non-interactive image galleries.
So in our work, we present GAN Explorer, an interactive tool for GAN exploration that
allows the user to assess capabilities and limitations of a GAN by “browsing” different
images that the GAN can generate. We then use the outputs of GAN Explorer to sample
even more photo-realistic images from the GAN, which we then organize into galleries
for further visual examination.
Due to the time constraint, I would like to direct you to our CHI video for more details
about our interface.
Here, I will focus on how we evaluated our
interface and our method in three empirical user studies.
In the first study, we show that our interface enables users to explore a GAN.
In the first study, we recruited 367 participants on Amazon Mechanical Turk, and asked them
to use our interface to select photo-realistic images generated using an existing GAN model, called BigGAN.
Participants selected images from at least one of 10 BigGAN categories,
for example, poncho, cup, butterfly. We measured how participants used our prototype to explore the GAN.
The results of our first user study show that participants were able to select over 10,000
photo-realistic images by interacting with different tools in our interface.
In study 2, we validated that images participants selected in Study 1 were actually photo-realistic.
We conducted another Amazon Mturk study with 1,622 participants and asked them to select
only photo-realistic images from an image gallery consisting of randomly sampled images from study 1.
The participants used a custom labeling tool that we developed for the study.
Our results suggest that our interface could support exploration and selection of photo-realistic
images from GANs. However, we also noted that participants often disagreed on what makes
a GAN-generated image photo-realistic. So, we visually examined images that majority
of participants labeled as photo-realistic and used them in our next study
where we wanted to compare photo-realistic images that participants selected using our
tool against the image galleries generated using existing methods.
However, crowdsourcing photo-realistic images from a GAN is not scalable.
Instead, we used a Markov Chain Monte Carlo method to sample and generate 25,000 images
from the posterior probability distribution of latent vector z given photo-realistic images
participants selected in study 1.
We then compared our method with the baseline sampling method that BigGAN authors used to
evaluate their model, where they sampled a latent vector z from a truncated normal distribution
with 4 arbitrarily selected thresholds.
In study 3, we recruited another 1,000 participants and asked them to select photo-realistic images
that occur only once in an image gallery using the same tool as in study 2. Only this time
we asked them to select images from galleries generated using our method and the baseline at four thresholds.
We primarily measured the number of photo-realistic images per Method, but we also considered the effect of Category.
Using visual examination, we found that our method consistently sampled more diverse galleries
of photo-realistic images, which you can see on the left compared to the baselines on the right.
We then ran the Align Rank Transform to analyze the data and performed a pairwise contrast test.
Our tests found a significant main effect of Method on the number of selected images
and our pairwise tests found a difference between our method and at least one baseline.
Not surprisingly our test could not find a significant effect of Catagory.
In conclusion, we showed that GAN Explorer enables principled qualitative GAN exploration
via interactive visual examination.
We also showed that our sampling method enables quick creation of many diverse, photo-realistic
image galleries to support qualitative evaluation of GANs.
For future work, we plan to expand the number of GAN exploration tools in our interface.
We will apply our method to other GAN models, including models that generate outputs other than images.
Our work will inform the design of a comprehensive interactive GAN diagnostic tool.
To demo GAN Explorer, please follow the link on this slide.
Thank you very much for your attention.