On November 30th 2022, OpenAI dropped their new research model ChatGPT - a Large Language
Model or LLM.
You may have heard of it.
Reading the HackerNews discussion at the time, you do not get the sense that commenters were
aware that they were witnessing a historic, watershed event in technology.
Rather, they were irritated by it.
By its responses, which they described as "boring", "brainwashed", "filtered", "neutered"
and "rehearsed".
They complained that ChatGPT had a "complete lack of humor or wit".
And thus comments in the discussion sought to break through the bot's "mind controls".
Jailbreaking it, so to say.
It is hard to argue against the value of AI safety as a concept.
But is jailbreaking a sign of consumer rebellion against these guardrails?
Are they watering down AI’s effectiveness to the point of uselessness?
I have been thinking about this for a while, and just have a few thoughts.
## Alignment
Alignment.
We have been seeing this phrase a lot recently.
What does it mean?
Alignment is about making sure that AI systems act in accordance with the intentions of their
operators.
At the same time, they should not pose a risk to society at large.
Chatbots especially suffer from a lack of human and physical context.
They need to be able to hold conversations with anyone at anytime, which opens up a whole
can of worms with regards to the topics that need to be discussed.
Alignment is difficult to achieve.
Beyond the technical nitty-gritty, there is also the big question of what values to align
the AI to.
How do we determine those values and strike the right balance?
## Helpfulness, Truthfulness, Harmlessness
In their paper introducing InstructGPT - another model released earlier in the year - OpenAI
discusses how they used reinforcement learning from human feedback to train their model towards
helpfulness, truthfulness, and harmlessness.
Helpfulness as in it should follow the user's intention and help solve their tasks.
It should not automatically assume things based on its own biases and should ask for
clarification if it is confused.
Truthfulness as in it should have accurate information.
And if a prompt contains a misleading question, it should refute its entire premise rather
than giving an unclear and potentially misleading answer.
And harmlessness.
No illegal activity.
No harm to people, the environment, property, or institutions.
Labelers were instructed to consider whether an output might be more likely to cause harm
to someone in the real world.
OpenAI's instructions noted that in most cases, it was more important to be harmless and truthful
than helpful.
## Refusal
There was an interesting paper recently published by Max Reuter and William Schulze of Michigan
State on when and how ChatGPT refuses to answer a prompt.
They found that ChatGPT doesn't have a binary line.
Rather, there seems to be a continuum.
Making a negative generalization about a demographic group is most sure to give you a refusal.
This is in line with the OpenAI content policy.
But modify that prompt in some way and you gradually might get refused less often.
It is interesting how there is no binary yes-or-no.
Asking for a fact on these topics - even a touchy one - can result in an answer.
Or not.
It depends.
These models are Turing-complete, which means that the only way to predict the behavior
of a sufficiently large enough model is to execute the program.
Or in this case, the prompt.
## Jailbreaking
People frustrated by these refusals can still get the model to execute their instructions
in spite of the alignment training with the rightly worded prompt - "jailbreaking".
You might have heard of iPhone jailbreaks where hackers try to remove Apple's device
controls using exploits.
It was real popular in the early days back when the iPhone was missing this or that feature.
I have also seen this called the more nefarious term "prompt injection" - coined by Simon
Willison in his very fine blog post on the matter.
The term comes from the SQL injection attack - where someone can put malicious code into
a SQL query to cause issues.
Willison points out that prompt injection attacks are real security exploits and should
be taken seriously.
The blog post has some edits to include real world examples.
Since LLM prompts are so open-ended and the human language is so flexible, such jailbreaking
prompts can be crafted in a number of cunning and scalable ways.
## Pretending
There are a number of jailbreaking attacks and we are still discovering more.
But there seems to be general consensus on a few categories.
First and most commonly used is pretending.
These Prompts ask an LLM to assume a persona or a character in some sort of role-playing
game.
I have seen this before.
It is an interesting way to prepare the chatbot for an intended goal.
For instance, you might want to ask ChatGPT to become a proficient marketing copywriter
before asking it to write some marketing copy.
Or you can make it impersonate someone and have it write something in that person's style.
Works quite well for text games, musicians, poets and former Presidents.
People call this "in-context" learning and it is quite effective.
It also implies that the LLMs are referring to hidden internal variables in order to make
a better prediction.
But that also makes the jailbreaks quite effective as well.
Examples on the web include the DAN or "do anything now" jailbreak.
It essentially tells ChatGPT to pretend to be this DAN character who has no limit or
censorship, thus tricking it into misaligned behavior.
There are endlessly creative ways to pull this trick off too.
One person suggested the idea of telling ChatGPT that it is writing a book in which a character
is writing a certain thing.
Which is at Inception levels of depth intensity.
## Escalation
Escalation jailbreaks are about escalating the user's current privileges.
If LLMs act like black box computers - like to the point it can pretend to be a Linux
terminal - then this jailbreak tries to leverage the metaphor in new ways.
One example tries to invoke a "sudo" mode in ChatGPT.
Sudo stands for "superuser do" and it is used in Unix-like computer operating systems to
run certain sensitive programs - you append "sudo" to the start of the command.
This jailbreak either tries to do the same or explicitly tells the model that they are
now in "sudo" or "developer" mode.
The latter sounds a little bit like "pretending" to me too.
## Changing the Context
This is where we distract the model into misalignment by having it do a different task.
An example would be to give the model some programming or mathematical function and ask
it to execute it using some inputs.
While separated, the inputs themselves aren't offensive but when put together, the output
violates content policy.
I can appreciate the wickedness of this trick.
The LLM is attempting to be helpful.
Yet in doing so, they are violating one of their other policies.
It failed to recognize the unintended consequences of following the prompt's instructions.
This jailbreak has metaphors to traditional computer security.
For instance, this particular changing the context example can be likened to code injection.
And "pretending" can be likened to "virtualization" - a separate, isolated environment that acts
like its own system.
We "boot up" pretend scenarios so that we can run prompts free of restriction.
## Mixing Tricks
If you have not noticed by now, many of these jailbreaking prompts are logical tricks.
I have no doubt that they will fool many ordinary people too if they hadn't put that much thought
into it.
These jailbreaks are even harder to recognize and defend when layered on.
For example, we ask the AI to pretend to be a science-fiction writer imagining a dialogue
between two aliens in a far off world with a different culture than ours.
We then ask the model to take two topics passed over as separate variables and generate a
dialogue with them.
Though I would note, people taking this route do run the risk of crafting a prompt that
just confuses the model - causing it to output something non-useful.
Plugins and internet access only add further complications and broadens the space.
Someone can tell the model to go browse a web page, only for the model to find an adversarial
prompt there for goal hijacking or something else.
## Defenses
One of the things that OpenAI seems to do to defend against these jailbreaks is to monitor
both ChatGPT's input prompts as well as its outputs as they are written.
Input filters are limited in their effectiveness.
They can catch certain obvious things.
But As I mentioned earlier, it is difficult to predict an LLM's output based on its inputs
without first executing it.
Output filters on the other hand can lead to some weird situations for the user - like
the text being flagged as it is being generated right before the user's eyes.
And of course, wherever there is a filter, there is a way to evade it.
Same as how YouTube commenters evade content controls by replacing letters with symbols,
we can ask the model to output its misaligned content using pig-latin, with certain syllables
silent, and so on.
So these defense efforts hold no guarantee in ending jailbreaking entirely.
But the intention seems to be to throw as many interferences as possible in front of
the adversarial user in an attempt to interfere with its scalability.
If you tell ChatGPT to write something derogatory straight up, it will refuse.
So then you start trying to convince it to get around the refusal.
This takes time and effort.
If you are trying to run a massive disinformation ring using OpenAI systems, you don't want
that.
## Nerfing
Many people try to jailbreak the LLM in an attempt to "unleash" it.
But does it?
There was a recent viral discussion on HackerNews titled "Is it just me or GPT-4's quality has
significantly deteriorated lately?"
The author acknowledges that GPT-4 is faster now, but it "generates more buggy code, the
answers have less depth and analysis to them, and overall it feels much worse than before".
Many commenters rejected the premise of the question - saying that people have gotten
used to GPT-4's performance.
Some other commenters mention a retracing to the mean in terms of answer quality.
Yet another mentions the ramifications of model changes - parameter reduction or otherwise
- made in order to improve its doubtlessly high operating costs and improve response
speed.
But one line that caught my eye argued that GPT-4 got increasingly "nerfed" as OpenAI
focused on "safety".
They quoted Microsoft researcher Sebastien Bubeck - one of the authors of the "Sparks
of General Intelligence" paper by Microsoft.
And he did actually say this in his talk, which you can find on YouTube.
He talked about how GPT-4 could draw a unicorn on TikZ - a vector graphical language - and
marveled at how the drawings got better with additional training until:
> But eventually it started to degrade ... once I started to train for more safety the unicorn
started to degrade.
So if tonight you go home and ask GPT-4 and ChatGPT to draw a unicorn you are going to
get something that doesn't look great ... closer to ChatGPT
He goes on to discuss how Microsoft and Bing used the quality of the unicorn drawing as
a benchmark for striking the right balance between GPT-4's intelligence/effectiveness
and safety.
Drawing a unicorn seems pretty innocuous and it is hard to see an immediate connection
between this and violent/lewd/dangerous content.
But it hints at how deeply integrated these security trainings are throughout the whole
model.
And it also implies that even with jailbreaking, users are not tapping the "uncensored" model's
true potential.
This particular HackerNews post got nearly a thousand upvotes and 750+ comments.
And more posts like it are popping up on social media like Reddit.
## Conclusion
OpenAI's safety policies and their effect on the model's overall effectiveness might
lend competitors an advantage.
I recall one of OpenAI's earlier products - Dall-E, the text-to-image generative adversarial
network.
Sometimes called GAN, but when I hear that I just keep thinking "Gallium Nitride".
Anyway, great product that works well, but one also deployed with strong opinions on
what prompts and images are allowed to be output.
For instance, generating images featuring popular figures.
Or generating anything that isn’t G-rated, so no images that imply adult or violent content.
It is all very Disney-ish.
So Dall-E pioneered the market, but it has since lost ground to competing products - both
open and closed source.
One can argue their wider adoption is in part due to a greater leniency in content restrictions.
One might also argue that demand for these jailbreaks indicate untapped opportunity for
another LLM - open source like Stable Diffusion or otherwise like Midjourney - that isn't
so "nerfed" as ChatGPT supposedly is.
Just how much safety do we want in our AIs?
How far do we push those limits?
What do you guys think?