This paper presents a natural language code synthesis tool
and results from a user study of this tool that provide key insights into the user experience
of developing code with large generative language models.
Recent large language models have shown the ability to perform a wide range of tasks.
For example, they can transform natural language requests into code like Python, HTML or JavaScript.
These capabilities have led to new tools like OpenAI's Copilot and Codex.
Recent work suggests
that these new AI assistants and cogeneration capabilities may be useful in software development.
This work examines the experience of using an LLM-based code synthesis tool
in which users can issue specific natural language requests, such as "Make an OK button." We developed
a tool GenLine backed by a large generative language model in several tasks specific prompts.
We tested GenLine on two tasks
with 14 study participants.
Our GenLine tool transforms natural language instructions into code.
In this example, the user types and natural language command to make a green square.
The combined prompt is sent to the model, which returns a set of responses.
In this example, the result shows the code and the rendered HTML.
Behind the scenes GenLine invokes a prompt designed to produce HTML
the prompt consists of two pieces.
The user provided input and the prompt prefix that primes the model to generate HTML.
In our interface double brackets are used to delineate the input for the model prompt.
Clicking the run button or control enter requests a result from the generative language model.
Users can also iterate on the input to refine the desired output.
Users can navigate through the model's outputs or accept the result to overwrite the original input.
Users can also mix natural language in code.
GenLine's predefined
prompts are specialized for coding tasks, such as adding IDs to HTML or generating JavaScript.
The output length is capped at a fixed amount.
To continue the code users can click the ellipsis icon.
To better understand the user experience of natural language code
synthesis with large generative models, we conducted a user study of GenLine.
In our study, 14 participants with differing levels of front end programming experience
completed two front end tasks, creating the Google search homepage and building a flashcard app.
Our results indicate that while this form of natural language code synthesis can sometimes
provide a magical experience,
participants faced a number of challenges accepting results less than 50% of the time.
Participants felt they needed to learn
a new syntax to interact with the model effectively despite their input being in natural language.
For example, one participant remarked, "It didn't feel like natural language.
It was more like, what's the right magic phrasing to get the model to do what I wanted to do."
As an example of how this challenge could arise when one participant tried to center a button
on the screen, the model centered the text within the button instead.
Participants also demonstrated a wide range of variability
in the scope and specificity of their requests.
For example, to start building the flashcard app, one participant requested, "Create a flashcard app"
while another participant tried, "Create a text input with the label front."
Participants also face challenges in debugging model input when results were not sufficient.
We identified a variety of recovery techniques, including rewording the request,
expanding or decreasing the scope of the request and rerunning with adjusted temperature.
One interesting fallback strategy observed in four participants was to inject words and phrases
such as Hello World or a test as easier targets.
Given these common strategies, we introduced the notion of a fallback prompt
which could be automatically executed by the system in parallel with the code synthesis prompt.
Fallback prompts are intended to help users move forward with their task by producing additional
or different types of output.
An example is a prompt that replaces the user input into a simpler request.
Given that participants sometimes make rather large requests, like producing an entire flashcard app,
it could be useful to leverage the generative language model
to iteratively refine and reduce the request through a dialog.
For example, the system could invoke a fallback prompt that suggests subtasks for a broader task.
Participants also sometimes tended towards complex multi-step requests, asking for several components
of the flashcard app at once.
For such examples,
the system could use a fallback prompt that transforms complex requests into simpler ones.
Despite these challenges in the current prototype, participants had high hopes
for future iterations of code synthesis tools.
In particular using natural language code synthesis to perform
the equivalent of API lookups resonated with many participants.
This video introduced the GenLine tool for natural language code synthesis, which provides
inline access to task specific prompts run with the generative language model.
We also report results from our user study of GenLine describing the strategies
participants employed to create code, as well as the issues that they had with the model syntax.