- I dare you to try and read this sentence.
Pause the screen if you have to.
(playful music)
Did you get it?
If English is your first language,
you were probably able to figure it out pretty quickly.
So why were you able to read that,
and what's the point of vowels
if you can still read perfectly fine without them?
If texting is your main source of communication,
you probably know what I mean.
The reason we're able
to read the sentence at all is usually attributed
to psychology and our brains' super-cool power
to fill in the blanks.
But there's another reason,
and it has to do with English itself
or more specifically the amount
of information contained in the English language.
But what does that mean,
the amount of information contained in the English language?
Like, how do you even measure that?
I mean, if someone were to ask you
which of these contains more information,
a five-letter word or five coin flips,
how would you even begin to answer?
Well, Claude Shannon, a mathematician
who's often referred to as the father of information theory,
came up with the concept of information entropy.
You may have heard the word entropy before,
usually being used to describe a messy room
or ice cubes melting or a demon in a box.
In physics, entropy means the measure
of chaos or disorder in a system,
the lower the order, the higher the entropy.
The way we're going to use it in this video is related
but slightly different.
In information theory, entropy is the measure
of information in terms of uncertainty,
the higher the uncertainty, the higher the entropy,
the higher the entropy,
the more amount of information is in the system.
This was pretty confusing to me the first time I heard it.
So let's try and answer the question from before.
Which has more information,
a five letter word or five coin flips?
Say you have a coin,
and it's one of those funky double-head coins.
A really useful way to think
about how much information is in a flip is to ask,
what is the least amount of yes-or-no questions I can ask
before I know the state of my system?
The more questions I have to ask,
the more states it could potentially be in,
so the more uncertainty there is
about what it'll actually be.
This coin is always gonna land on heads, right?
Before you even look, you know it landed on heads.
You don't need to ask any questions.
There is zero uncertainty about it.
So this system has zero entropy,
which means it contains zero information.
Now, when I first learnt about this,
one thing I got stuck on with the exact meanings
of entropy and information,
they're related but they're not the same thing.
One way I like to think about it is
that entropy is the measure of uncertainty before the flip,
and the information is the knowledge you have
to gain after the flip.
So here, there was only one possibility,
so there was zero uncertainty before the flip,
and you gained no knowledge
by looking at the coin after the flip.
So the system contained zero information.
Now, if we replace our funky coin
with a regular fair coin, let's see what happens.
Now you have two options, heads or tails.
So how many yes-or-no questions do you need to ask
to know which way it landed?
Well, you could ask, is it heads?
And with one yes-or-no answer, you'll know.
So the entropy of a regular coin toss is one.
Now, if we have five coins,
each of the coins has an entropy of one.
So the amount of information in the system is five.
So the amount of information contained
in five coin flips is five bits.
Oh, yeah, bits is the the unit of information.
So how about a five-letter word?
We could just start randomly guessing words.
Is it dodge? Is it juice?
But that's a pretty terrible way.
Remember the entropy is the least amount
of questions we could possibly ask.
A good approach would be to go letter by letter
and ask questions which eliminate half of the possibilities.
For example, the middle of the alphabet is between M and N.
So we could start with,
is the first letter before N?
If the answer is yes,
we can eliminate the lower half of the alphabet.
If we divide in half again and ask is it before G,
and the answer is no, we can eliminate half again.
We keep doing this until we get to the letter we want.
On average, about 4.7 questions will be needed per letter.
5 times 4.7 is 23.5.
So the entropy of a five-letter word is 23.5.
A five-letter word contains more information
than five coin flips.
But does it really?
I mean, you got the sentence at the beginning
of the video without asking 4.7 questions per letter.
I hope.
This is a page from Claude Shannon's original paper
on the prediction and entropy of printed English.
The first lines are the original text
and the second lines are the participant guesses.
The dashes represent all the letters
that were correctly guessed on the first try.
And these letters are the ones
which took one or more guesses.
He found that 69% of the letters were guessed correctly
on the first try.
Here's another example which specifies exactly
how many guesses each letter took.
Note the larger amount of ones.
So what gives?
Well, when we guess the five-letter word, we are assuming
that English was a totally random sequence of letters,
which, as my third-grade teacher kept reminding me,
it isn't.
There's grammar, spelling, patterns,
things that make it less random,
like the fact that q is always followed by a u,
or that e appears a lot more than z.
In information theory,
these are called redundancies.
My friend Mark, who runs a channel all about words,
knows a lot of cool stuff about redundancies,
and we're lucky enough to have him share some
of that stuff with us, today.
- Thanks, Jade.
Yes, you're certainly right,
that English doesn't have the most efficient spelling system
and people complain about it all the time,
but the history of English spelling is moving
from higher entropy to lower entropy, and paradoxically,
some of the confusion is the result of that shift.
More and more patterns and information have been added
to make English easier to read.
In ancient Latin and Greek, for instance,
there were no spaces between words, no capital letters,
well, actually, only capital letters,
lowercase wasn't invented until the Middle Ages,
and no punctuation.
How easy do you find this to read?
Actually, if we go right back to the beginning
of the alphabet used by various Semitic peoples,
such as the Phoenicians, there weren't even any vowels,
like in the sentence Jade started this video with.
No problem, right?
Well, maybe a bit of a problem.
And how about this?
Because for a long time, there wasn't any rule
about which direction people wrote.
Sometimes words went right to left,
sometimes left to right,
and sometimes back and forth, in the same document.
But the thing is, English doesn't work as well
without vowels as Phoenician did.
And that's part of the story
of why languages such as ancient Greek, Latin,
and English have added so many extras to the writing system,
and there are elements of English spelling,
such as q always being followed by a u,
or h only following certain consonants
or having a silent e on the end of many words,
or double letters, that help us read a word,
even if we can only see some of the letters.
This type of redundancy,
where there's more information on the page
than is strictly necessary for comprehension,
can make English spelling seem complicated.
So, Jade, why does English have so many redundancies?
- Redundancies are really useful
for keeping a message safe from interference or noise.
Like when you're
in bad reception! (static crackling)
(loud rock music) Or when it's hard
to hear what the other person is saying!
(static crackling) In situations
where information can be lost, which, let's face it,
is a lot of the time,
(loud rock music) redundancies assure
your message can still be communicated.
So what's cool is that we can tailor the interplay
between entropy and redundancy,
depending on the message we want to send.
When texting a friend,
it's cool to have a bit of uncertainty,
because it doesn't matter too much
if there's a miscommunication.
So here, you can send a quicker message
with less information.
But with your boss,
you're probably better off taking the time
to send a clearcut email, spelling everything out,
because if there's any miscommunication,
well, there can be big consequences.
Basically, if you want to send a cheaper,
faster, more efficient message,
you'd send a message with higher entropy.
If you need to send a message where it's super important
all of the information gets across,
you'd send a message with more redundancies.
And you can always mix-and-match.
In sending messages to outer space,
it's extremely expensive
and super-vulnerable to information loss.
But you really don't want
an angry alien getting the wrong idea, either.
Oh, and in case you were wondering,
Claude Shannon actually did calculate the entropy
of English to be 2.62 bits per letter, on average.
So that's a bit about the math of the English language.
If you wanted to know more about the history
and how we got all these wacky rules in the first place,
Mark's got you covered.
- Thanks, Jade.
We're telling the story of English spelling,
from ancient Egypt and Rome
to inventive Anglo-Saxon writers, Vikings,
and snooty French scribes to a king's court
and an unsatisfying grocery shopping experience.
And along the way, we'll explain why spelling is so magical,
the etymology of diphtheria, and the real story
behind that troublesome G in GIF, or jif, whatever.
- Thanks for watching, guys.
I can't wait to find out if it's pronounced gif or jif.
So head over to the video we did
on Mark's channel Alliterative.
There's some really interesting stuff over there.
I'm back. That was super abrupt, right?
Yeah, I didn't really know how to segue into that,
but anyway, the point is that I'm back.
I'm back from film course. I'm back from VidCon.
And now I'm here and ready to make videos again.
I really missed it.
I really missed you guys.
So yeah, be seeing you shortly. Bye!
(lively rock music)