hi everyone today i'm going to talk about AI chains:
Transparent and controllable human-AI Interaction by Chaining Large Language
Prompts. This work is done when I was an intern at google research with Michael Terry and Carrie Cai
Since 2020, our NLP community has deployed this so-called large language models, and what’s
impressive about them is their ability to respond to arbitrary natural language prompts at run time.
For example, if I want to do English-to-French translation, I don’t need to train a separate
engine, but just feed into the model something like: Given the English sentence,
translate to French; And then English sentence, and then denote the starting point of French,
then the model can then continue the prompt with the French translation.
But users want to do more things than translation. Many of our user were google employees,
and they really wanted to use this same large language model for rewriting their short summaries
into friendly and constructive peer review paragraphs. The short summary might look like:
Sherry could improve her presentation skills. She has too much text on her slides. Her presentation
meanders from topic to topic without a clear structure. She also does not engage with her
audience when she presents. The model indeed can do rewriting, and it generates something for them,
but this generation doesn't really match with their expectation. Here, you can clearly see
this paragraph is pretty mediocre. It’s still mostly impersonal and does not provide concrete
suggestions. For example, it just says “you should have a clear structure” in response to “without
a clear structure” and if I got this feedback I would be like, well I still don’t know what to do.
Now, if you work on nlp or more specifically, text generators, then you probably can guess
what went wrong. Basically, there are some limitations in the modeling structure,
and the model doesn’t really know how to do reasoning, they start to get biased by their
own generations as the text becomes longer, etc. If you don’t work on these, don’t worry, because
end users also don’t know what I’m talking about! But they are the users of these models, so we
need to answer this question: ”How can we help end users interpret and improve model outputs,
when they don’t understand the underlying structure and cannot change the model?”
Our way of doing it is to build this kind of “AI chains”. Basically, we try to provide some more
explicit knobs to end users, by breaking down an original task into sub-tasks.
What’s the intuition behind this? If we really think about
many many tasks similar to this rewriting, the problem from the end user point of view is,
because this task has a lot of things going on, it’s hard to build an input to output mapping.
It’s not clear how they can help the model to know what presentation problems exist,
and brainstorm suggestions, and also do the rewriting.
But this mapping exist if we make the task smaller and more targeted. Instead of rewriting a full
paragraph, if we just ask for suggestions on a single problem of “does not engage”, then we get
very explicit suggestions like “ask the audience questions” or “walk around the room”, and then end
users can come in and say “I don’t like this because, I can’t walk around the room during
a pandemic, so I will keep iterating on it until I’m happy.” And maybe get things like “use humor”.
And then, if we pass the human-fixed suggestions to another large language model that specifically
focus on rewriting suggestion bullet points into friendly paragraphs,
then we can get a better paragraph, like here “You may want to ask more questions
to engage the audience. Humor always helps!” Packing them together, we have a chain. We ask
large language models to complete each sub-tasks separately, and use the output of one step as
input for the next, so the final result will get to aggregate the results from previous runs.
Our peer review rewriting task can be completed in three steps.
We build an interface for users to actually play with the Chain.
Pretending you are the peer review writer now, here’s how you're going to use the Chain.
In the interface, we will use this visual flow to track the underlying structure
of what large language model steps are involved and how the data are transformed.
Getting into our first step, given the original feedback, we ask the model to split
it into a list of separate presentation problems. The prompt is very similar to what we saw before,
though we make it look more structured in the interface.
Then we take this output, problem list, to the second step, And ask large language model
to ideate suggestions per problem. So here we can run the model on too
much text independent of other problems, and get suggestions like “Use more graphics”, refine them,
and when we are ready, and then move to the next problem of “Meanders from topic to topic”.
Then, we use both the problems and the suggestions as inputs in the third step
to compose the final paragraph so that all of the results from previous runs get aggregated.
And now this rewrite is much better. It now has praises like “oh your presentation was
so interesting”, and good suggestions like “it might be helpful to vary pictures with text”,
and “if it were me I would divide the sections.” If you think about this, it’s really amazing.
Arguably in all of these three parts we asking this nice large language model to
do something simple, so in a way we are repeatedly under-utilizing what it can do, but just by doing
this, the final result is also better. So it’s saying for these models, sometimes less is more.
Through a user study comparing Chaining to just the standard baseline without chaining,
we see our users like interacting with LLM through chaining. In particular,
they find chaining makes their interaction with LLMs more transparent,
because after the task decomposition, then they are more aware of what things the model can do.
They also feel that interacting and changing the output of each step helps them to steer the system
towards their final goal. One participant phrased it like this, and I really like it. They said,
”chaining makes it easier to fine-tune things…Too much freedom without Chaining is a curse.”
So Chaining seems nice, but how can we decompose an overarching task, such that each independent
run of the model will be effective and can contribute to the final goal? We design a list of
operations that can help better scope the subtask inputs and outputs when the original task is too
broad, and we demonstrate their usefulness on a wide range of use cases, including,
Diversifying topics for language learning flashcards,
Handling rare data formats in visualization code debugging,
And supporting branching logics in assisted text entry apps.
And across these use cases, we see that the concept of Chaining raises the ceiling for
what people can use large language models for, our paper has all these details so please do read it!
We also have a follow up work on helping people to prototype AI-infused application
with Chaining. If you are interested in how a UX designer would build this music chatbot,
please check out our Late Breaking Work called PromptChainer, where we show how
AI Chains lower the floor for end-users prototyping with large language models.
As a recap, AI Chain breaks down an original task into sub-tasks,
With this, we raise the ceiling of what the human can do together with LLMs, in a more transparent
and controllable way, and we also lower the floor for end-users prototyping with LLMs.
Most importantly, we achieve all of these just through interaction designs,
without having to change the model itself.
Please check out our paper and our demo video for more details. Thank you