Hello, I'm Michael Terry, and I will be presenting our case study examining the use of prompt
programming for rapid prototyping of AI.
Prototyping is essential to software development, but it’s hard to prototype AI.
Recent work with large language models, or LLMs, have demonstrated that these language
models can be quickly customized simply by carefully crafting the input sent to the model.
For example, to create a basic English to French translator, we can create the input
shown on the right.
When we provide that input to the model, it is likely to correctly translate the phrase
"How are you" to French.
The key insight here is that these natural language prompts – that is, the text provided
to the model – allow people to rapidly prototype new models on the fly.
To this end, we’ve created PromptMaker, a platform for rapidly prototyping new ML
models using prompt-based programming.
With PromptMaker, a designer or product manager can come up with an idea for an AI-based feature,
write a prompt, and then immediately test it on a range of inputs.
How does “prompting” affect ML prototyping?
To help answer this question, our case study reports results from interviewing 11 industry
practitioners using PromptMaker.
These practitioners include designers, program managers, content strategists, and front-end
software engineers.
What we found was that: PromptMaker dramatically reduced ML model
prototyping time (months to minutes/hours) It enabled non-ML experts to be more self-sufficient
It provided early intuitions about what might be possible
And prompts themselves served as boundary objects to ground communications
Let's dive into these findings, starting with the time required to prototype AI.
Often, the prototyping process for AI-based features can slow down when a designer tries
to go from an idea to a functional mockup For example, this participant said, "You have
a pen and paper idea, you’ll play with it using Sheets or Slides or pen, paper, scissors…as
soon as you have to write code, the process slows down, someone has to write a backend
and server, everyone needs to have a Linux box."
But now, by using PromptMaker, this team notes: "We thought about natural language queries
for [a product].
15 minutes later we had a working example of natural language to structured queries."
In other words, models that would previously take months to prototype can now sometimes
be developed in minutes.
These abilities also reduce the reliance on other experts.
For example, without PromptMaker if you don't have the expertise to build a model, you must
rely on others to build one for you.
For example, this participant said "[We spend] a long time trying to find the ML researchers…We
don’t have a lot of ML engineers on our team."
But now, it’s possible to start experimenting without needing to have a machine learning
expert or software engineer on hand.
This can be quite empowering to people like UX designers, who can now create functional
AI prototypes themselves.
Rapid prototyping also helps gain early intuition about the feasibility of an idea.
If you don't have a lot of machine learning expertise, it can be hard to know what's possible
from a modeling perspective.
Instead, you must find experts to ask about the feasibility of an idea, as this participant
notes: "It was sort of a process of interrogation -- not of the model itself, but of the people
who had the ability to understand it or created it."
But with promptmaker, participants indicated they could get a direct sense or intuition
for what’s possible – for example, this participant said that "Being able to play
around with [the model] directly and…get a sense of what things are easy and what things
are hard…just so much more useful than a wishful thinking sample dialog."
The prompts also help communicate an idea.
Without prompts, participants noted that communicating an idea in detail requires significant time
and documentation.
With prompt programming, a prompt can serve as a boundary object, as this participant
notes: "I was referencing something, but he [SWE] could then use it.
It was a common material that we were both using…really important in terms of increasing
that velocity."
Our interviews suggest that PromptMaker and its prompts are supporting practices similar
to “sketching” or prototyping as found in other contexts!
Similar to prototyping, prompt-making is fast (...), accessible (...), grounds communication,
and affords lightweight testing.
At the same time, participants also faced several key challenges in developing prompts
Specifically, they had challenges in: Reverse-engineering prompt designs
Sourcing examples Prompt debugging
Evaluation To go into more detail, prompt engineering
can be deceptive: It seems like you can use any phrase just as if you were talking to
another person.
However, that's not quite the case.
It's possible to have similar phrases, but different outcomes from the model!
Here’s an example where similar phrasing in two different prompts can yield very different
outcomes.
In the first example, we ask the model to use a word in a sentence.
The second example requests the same outcome, but is more likely to produce the desired
results.
These differences are due to what training data the model has seen.
Given this, users need to work backwards, by first considering what phrasings are more
likely to mimic text in the training data.
In short a key challenge in prompt engineering is in developing an accurate mental model
of how the model works so you, as a user, can “reverse-engineer” a prompt design
that works.
There is also a need to come up with multiple examples so the model doesn't overfit.
But coming up with multiple examples can be challenging, and it can be difficult to know
whether the examples are sufficient to produce the desired results.
To address this problem, most participants came up with examples on their own, and some
searched the web to find content.
One participant recruited a collaborator to broaden their set of examples, while another
participant sourced additional examples from the model itself by picking good results from
initial model output, adding them to their prompt, running
the model, and then repeating this process.
Because a generative model may produce different results on each run, it can be challenging
to systematically evaluate whether a particular prompt is better than another.
Here’s an example where someone wants to prototype a model that uses a given vocabulary
word in a sentence.
Say they see an improvement in output.
With just a single run of the model, it is challenging to know whether this is due to
chance or the actual prompt design.
At present, prompt testing is quite ad hoc.
Additional tools would be useful to more reliably evaluate different prompt designs.
So to summarize what we saw, we found that prompting made it possible to rapidly prototype
ML: It dramatically reduced ML prototyping time
from months to minutes It also empowered people like UX designers
and product managers to directly sketch out ML models
It enabled them to be self-sufficient And the prompts helped grounded communication
between team members
At the same time, participants noted several challenges while making prompts.
They found that they needed to reverse-engineer their goals when designing a prompt..
They also found it difficult to create good examples that could generalize.
And because the LLM could produce different output on each run, it was also challenging
to systematically debug prompts
Thank you for your time.