Okay.
I want you to check this out
because it represents a big challenge
for large language models like GPT-3...
and now GPT-4.
But it is not code.
It is a list that countries around the world
are grappling with.
Before we get into the problems
with large language models, let's review at a basic level...
how they work.
You've probably heard of ChatGPT.
It's not really a model
but an app that sits on top of a large language model.
In this case, a version of GPT.
One thing models like ChatGPT do
is natural language processing.
It's used in everything
from telephone customer service to auto completion.
GPT is like a used bookstore.
It's never left the room
and only learned through the books that are at the store.
Essentially, these large language models...
they scan a lot of text and try to learn a language.
They can check their process by covering up the answers...
and then seeing if they got it right.
Then they can use that knowledge to recognize sentiment...
summarize, translate
and generate responses or recommendations
based on the analyzed data.
And yes, ChatGPT wrote that last line.
You've got to do that in videos like this.
This is an amazing ability
but that's because it's read a lot of stuff.
You can ask ChatGPT
to rephrase something to Shakespeare
but that's because it's read all the Shakespeare
and that is where my waste of paper comes in.
Actually, I want to show you something.
Yeah? Good?
Okay.
So this is a print out of Common Crawl
from 2008 to the present.
Common crawl basically means that they go over all the websites
and index them.
And on this list, they put every language
that they think that they've indexed.
Here, you notice right away all the English.
Every crawl is like more than 40% just English.
German: DEU.
See the indexes here...
you know, it's about 6% every time
which doesn't sound like a lot, but it's kind of a lot.
But look here, 2023.
FIN: Finnish. Lot of pages.
But it's just 0.4% of the entire scan.
This bookstore, it's got an inventory problem.
All of the focus is on only a very small set of languages.
There was a paper that stated that of the 7,000 languages spoken globally
about 20 of those languages
make up the bulk of NLP research.
Okay, so let's back up a bit.
This is Ruth-Ann Armstrong.
She's a researcher who I interviewed and she's doing something
that a lot of researchers are trying to do...
make new data sets.
Those 20 languages fall into our category called
high-resource languages
and the others fall into a category
called low-resource languages.
Those low-resource languages
don't show up on the Internet as text as much
which means they don't make it into language datasets.
They become unintelligible to the AI.
Imagine our used bookstore again.
It has a ton of Dan Brown books
or James Patterson or Anne Tyler.
This is like English and German and Chinese.
The high-resource languages.
Then there are the rare books.
These are the low-resource languages.
So, many models just don't know as much about them...
or have anything at all.
I’m someone from Jamaica.
The language primarily spoken in Jamaica is English...
but we also speak a Creole language called Jamaican patois.
Armstrong and her coauthors wanted to create a dataset
that can explain this largely spoken language
but they weren't trying to generate texts like ChatGPT.
Instead, they wanted their model to understand it.
In this case, to do that, Armstrong went through
a bunch of examples of Jamaican patois and lined them up.
Two columns.
And she labeled whether the statements
entailed or agreed, contradicted, or were neutral.
You can try it in this one.
A has a fever.
B has a high temperature.
So it’s entailment.
They agree.
Try this one: Entailment or contradiciton?
Contradiction.
One more.
Neutral.
The two statements don't really relate.
She did that for almost 650 examples.
You can probably see that this was a ton of work.
And Jamaican patois is not on my big list of
Common Crawl languages.
I also talked to some Catalan researchers...
who are trying to evaluate how well these big language models
do on stuff like Catalan.
It is the most spoken in this autonomous community of Spain.
In GPT-3, the percentage of English words is 92%.
For German, there's 1.4% words.
Spanish appears in 0.7%.
And finally, Catalan.
The amount of Catalan words in the whole training set is 0.01%.
And it still performs very, very well.
So the problem here is a little bit different, right?
They've got some Catalan in the dataset.
Common Crawl says Catalan is 0.2335% of their survey.
Not a lot, but some.
In the big company models like GPT-3 and presumably GPT-4 in the future
were proven to do pretty well on little data.
For example, the research team got GPT-3
to generate 3 Catalan sentences.
And then they mixed them up with real sentences.
Three native speakers then evaluated them.
So, that was our test.
And their results were very good for the machine.
But there is still a catch.
It performs reasonably well.
But it's worth it to build a language-specific model
that has been specifically trained and evaluated for that language.
So the problem here isn't performance
it’s transparency and it's the amount of data.
I mean, Common Crawl says that they indexed...
millions of examples of Catalan words.
But GPT-3 says that they only read
about 140 pages of Catalan.
Imagine like a novella.
It's a problem being dependent
on the performance or even the goodwill
of a few institutions or a few companies.
You can easily imagine a world where one of these companies
just cuts out Catalan.
The same way Catalan News complained Google was cutting out
Catalan links in searches.
Common Crawl is just a percent of what
GPT-3 was trained for.
We don't know the details about GPT-4.
And that means a lot of other stuff
went into this language model that we just don't know about.
Right now all these bookstores are actually run by
Meta or Microsoft or Baidu or Open AI or Google.
They decide which books go in there
and don't tell anyone where they came from or who wrote them.
Some people are trying to build a library
next to the bookstore.
This is Paris, where the French have a supercomputer
that wasn't being used a lot.
It’s like almost down the road
and I was discussing with the people...
who built it and they're like, “Nobody uses this GPU.”
Basically, what can we do?
Thomas Wolf is a co-founder of Hugging Face
which is like a hub for AI research on the Internet
and they ended up working on Big Science’s BLOOM...
a project to create an open-source multilingual model.
And the more we thought about it
we thought it's also a lot better, in fact, that we trained it
in a lot of other languages, not just English.
And if we try to involve many people
and so it started from a small Hugging Face project
to become a very big collaboration.
Where we tried to open this to everyone.
They basically went down the Wikipedia list
of most spoken languages and covered those.
But also added low-resource languages when possible.
So we have very, very low-resource languages there.
Mostly in African languages.
And so here to gather the data there
what we decided was to partner
as much as possible with local communities
and ask them basically what they thought were good data
and how we could get it.
As importantly, we know where the data comes from
and how it was obtained.
That's the difference in open-source.
You know the books in the library.
All right, let me find English.
Okay, so let's being honest as an English speaker...
I'm kind of the target audience
for these big companies in these big models.
English represents
more than 40% of the Common Crawl
but there are reasons for even the target audience
to want all languages to be well represented.
I am an English speaker but I have my Jamaican accent
and I remember that...
initially like when Siri came out, I had a harder time using it
because it couldn't understand my accent.
So expanding even the training dataset
for voice assistants include...
more accents has been helpful.
So imagine what would happen
if we tried to expand another piece of that.
We're building technologies for more languages as well.
So if you want to have this model everywhere
you need to be able to trust them.
So if you trust Microsoft, that's fine.
But if you don't trust them...
yeah.
It's our language.
So we speak— we are Catalan speakers.
So whereas because of a small language
or of a moderately small language
because you may have languages that have a...
a sizable amount of speakers in the real world
but that have very, very little digital footprint.
So they are bound to just...
disappear.