Our paper "Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm" explores
new methods of prompting autoregressive language models.
We use GPT-3 as a case study.
It is a state of the art language model released by OpenAI last year.
It is an autoregressive model which takes an input prompt and predicts a likely continuation.
It's the biggest neural network trained to date, at 175 billion parameters.
When testing the performance of GPT-3, researchers usually use a method called few-shot prompting,
where a number of solved examples of the benchmark task are provided in the prompt before an
unsolved task the language model is expected to complete.
For example, for an English to French translation task, the examples are pairs of English phrases
and their correct French translations.
The target task is an English phrase at the end of the prompt.
GPT-3 understands that the pattern should continue and a French translation should follow.
This is the method the OpenAI team used in their paper, "Language Models are Few-Shot
Learners" by Brown et. al.
This ability is sometimes called meta-learning, alluding to ability to perform tasks just
from seeing similar examples in the prompt instead of using fine-tuning.
However, few-shot prompting is not the only method of controlling GPT-3 and measuring
its performance on benchmarks.
We argue that other prompt formats can be more effective, including prompts with no
examples of the task at all, usually called zero-shot prompts.
Our basic argument is as follows.
GPT-3 was trained on a massive, varied corpus of "language in the wild": meaning text from
the internet and books.
Because GPT-3 was trained on real human data, its idea of what should come next is very
similar to our ideas about what makes sense.
So, with a few caveats, to prompt GPT-3 we can think about which prompts we would expect
a *human* to complete in a way that accomplishes the desired task.
So most effective prompts resemble the training distribution, which is largely made up of
natural language and usually not in contrived lists like the few-shot format.
Though few-shot is a very powerful technique for task specification, just as demonstrations
are valuable for conveying a task in human communication.
Other formats which simply describe the desired task instead of showing examples can be just
as effective.
We demonstrate the success of 0-shot prompt using a French to English translation benchmark.
OpenAI's results on the benchmark, and in fact all benchmarks they tested, suggest that
performance increases with the number of examples provided in the prompt.
64-shot prompts performed significantly better than 0-shot prompts.
We reran the translation benchmark with an alternative zero-shot prompts, including one
that used a colon to indicate which sentence was French, and a colon to indicate the completion
should be English.
This "simple colon" prompt and other zero-shot prompts we used are more natural to humans
and also more effective at indicating the desired task to GPT-3.
We also tested a few-shot version of the simple-colon prompt which includes examples.
We tested two sizes of GPT-3, using OpenAI's API, Babbage and Curie, with zero-, one-,
and 10-shot versions of OpenAI's prompt and our simple-colon format.
For OpenAI's prompt, similar to in the paper, 0-shot scored the lowest and 10-shot the highest.
For the simple colon prompt, 0-shot has equivalent performance to 10-shot, and outperforms OpenAI's
10-shot prompt.
GPT-3's success with zero-shot prompts shows there is some ambiguity with the phrase "meta-learning",
which is acknowledged in OpenAI's paper: is GPT-3 actually "learning" the function demonstrated
by the examples from the examples, or do the examples just communicate what the intended
task is, which GPT-3 already knows how to perform?
Our zero-shot prompt produces the same performance as the few-shot prompt, suggesting that the
examples in the few-shot prompt are not used to teach GPT-3 how to translate; that ability
was learned during training.
Of course, it is infeasible to learn to translate from just a few examples.
The examples here are more suitably described as "task location" rather than "meta-learning".
Some takeaways from our paper.
Further thought is required to properly evaluate large language models, as they can be extremely
sensitive to the prompt format used
As language models become more powerful, new prompt formats will become practical, including
ones which closely imitate human communication
This is a new paradigm for human-computer interaction in which anyone who is fluent
in natural language can be a programmer
Thanks for listening