Hi, my name is Vivian and today I am presenting a paper titled “Design Guidelines for Prompt Engineering Text-to-Image Generative Models”
So these models are a new and powerful way to generate visual artwork. A model can simply take in a text prompt and translate it into an image. and because text is free-form and open-ended, the possibilities for image generation are endless.
However, this also generating can easily become a brute force and trial and error kind of design process, which can feel very random and unprincipled.
In this paper, we conduct a study that explores what prompt keywords and model hyperparameters can help users produce coherent outputs.
Across 5 experiments, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. We look at over 5000 generations, spanning over 50 subjects and 50 styles.
In the first experiment, we test different phrasings of the prompts to see how different orderings of keywords, function words, and filler words affect generation quality.
We wanted to understand if users would need to try many different permutations of the same prompt to find the best way to word one.
For example, would a generation from “an ocean in the disney style” be significantly different from a different permutation fo the same words “an ocean in the style of Disney”?
So to understand this, we generated 12 subjects and 12 styles with 9 different prompt permutations and presented these grids to annotators.
From the annotation results, we found that there was no significant difference between the quality of the images generated from these 9 prompts.
And so from this experiment, we concluded the following design guideline which is that when picking the prompt, focus on subject and style keywords instead of the connecting words.
For the second experiment, we tested different random seeds to understand if different seeds paired with the same prompt would yield significantly different generations.
The motivation behind this was to understand whether or not users would need to try multiple seeds before moving onto new combinations of keywords.
We had a similar setup to the one before, where we generated 12 subjects, 12 styles but now with 9 different seeds.
After presenting these grids to annotators, we found that there was a significant difference. The number of generations judged as outliers was significant when compared to the number of generations deemed not outliers.
And so this implies that the stochastic components of the generation process can significantly vary the quality of generation. And the design guideline that follows is to generate between 3 to 9 different seeds to get a representative idea of what a prompt can return.
For the third experiment, we wanted to investigate on average how many iterations are needed to get a favorable result. Our specific research question was: does the length of optimization correlate with better evaluated generations?
To study this, we tested 6 subjects across 12 styles iwth constant seed and one variety of prompt permutation. We saved images at every 100 iteration from 0 to 1000 iterations and we had annotators annotated the rows for which generation they most preferred
We found that the 200th, 100th and 500th iteratons were chosen as the most preferred and this demonstrates that a higher number of iterations did not necessarily correlate with a more desirable generation.
We concluded from this experiment that when generating, using shorter lengths of optimization between 100 and 500 iterations is sufficient.
In the fourth experiment, we wanted to see if the model would perform equally well across a breadth of styles.
We also wanted to understand if the framework would perform differently for different traditions of styles.
To rigorously investigate this, we looked at three “partitions” of styles: abstract versus figurative, Western versus non-Western, and styles partitioned by time period (digital, modern, and premodern).
We generated from 51 styles that balanced these different partitions across 12 subjects. Annotators were instructed to judge how well each style was represented in the generation.
From our annotator results, we found that the model performed better on some styles than others.
The model performed significantly better on figurative styles as opposed to abstract styles. And differences between digital, modern, and premodern art styles was also significant.
However, the difference the difference between Western and non-Western styles was not. And we concluded from this experiment that when choosing the style of the generation, one can feel free to try any style, no matter how niche or broad.
And in addition to this quantitative analysis of the generations, we also analyzed the best and worst performing styles of each partition and summarized success and failure modes.
Success modes pertained to color, technique, relationships in space, and motifs. For example, success in terms of color meant stereotypical color palettes, correct textures, and the correct contrasts.
Successful in terms of technique meant that the generations exhibited choices of line, texture, and elementary brush strokes that were congruent with the original styles.
The failure modes that we found were interesting as well, as they were very unique to text-to-image generation. For example, one was that the prompt could be misunderstood owing to the multiple interpretations of the original prompt.
For example, generations showed that the model generated for a different parsing of the prompt or generate for a different meaning of one of the keywords in the prompt.
Another failure mode was when text would emerge in the generations potentially as a compensatory technique on the model’s part to optimize towards the prompt.
Our last experiment focused on the interaction of subject and style in this experiment. We wanted to understand: do certain categories of styles, such as abstract or figurative styles, perform better on certain categories of subjects, such as abstract or concrete subjects?
To study the effect of interaction of subject and style, we generated from 51 subjects, 31 styles. And what we found was that the top ten subjects were all concrete subjects in nature.
And they were subjects that were universal across most cultures: for example oceans, forests, eyes, and birds. Additionally, we found that when we compared the abstractness of the noun to the generation quality, there was a moderate to strong positive association, meaning that on average concrete subjects tend to do better.
And in running a two-way ANOVA test, we also found that subject, style, and their interaction all have a significant effect on the rating of the generation.
Again we also summarized success and failure modes this time pertaining to subject representation in the generations.
We found that one prominent success mode was that for many subjects, the models could access and apply symbols.
On the other hand, a common failure mode, and particularly for abstract subjects, was that sometimes the subject would simply drop out.
One of the most poorly rated of the subjects was website, because websites and digital media are anachronistic subjects to many modern and all premodern styles.
So in sum, for Experiment 5, we concluded the following design guideline: when picking the subject of the generation, pick subjects can complement the chosen style in level of abstractness and relevance.
We reiterate here the design guidelines that we provide for text-to-image prompting. We highly encourage you to checkout the paper for more details and thank you for your attention.