now that we've learned about entropy
thermodynamics and statistical mechanics
were ready to talk about Claude
Shannon's formulation of information
which is now called Shannon information
theory and turns out to be an important
concept in many areas of complex systems
research it's definitely something you
should know about Claude Shannon was a
mathematician working in the 1940s at
Bell Labs which was part of AT&T his
focus was on a major question for
telephone communication how to transmit
signals most efficiently and effectively
across telephone wires in developing a
mathematical solution to this problem
Shannon adapted Boltzmann's statistical
mechanics ideas to the field of
communication and use these ideas to
define a particular somewhat narrow but
extremely useful notion of information
in Shannon's formulation of
communication we have a message source
this is here a black box that emits
messages but you can think of it as
someone you're listening to over the
phone the message source emits messages
for example words to a message receiver
that might be you listening to say your
mother talking to you on the phone a
message source more formally is a set of
all possible messages that this source
can send each with its own probability
of being sent next and a message could
be a symbol a number or a word depending
on the context in our examples they'll
mostly be words the information content
in Shannon's formulation is a measure of
the message source which is a function
of the number of possible messages and
their probabilities informally the
information content H is the amount of
surprise that the receiver has upon the
receipt of each message so let me
explain that by giving a couple of
examples in my book complexity a guided
tour I used the
examples of my two children when they
were much younger as you can imagine
they're now teenagers and they aren't so
happy with being examples in my book so
I won't name them here but you can
imagine any particular one-year-old
who's the message source and this
particular one-year-old
is talking to his grandmother on the
phone but all he can say is one word da
da da da he says it over and over again
so his messages consist of one word da
with probability one that's all he ever
says there's no surprise there for from
grandma's point of view she always knows
exactly what the next word is going to
be and therefore if there's no surprise
there's no information content so the
information content of this message
source that is the one year old is equal
to zero bits information content in
Shannon's formulation is measured in
bits
in contrast think about the one year
olds older brother who's a three year
old the three year old is able to talk
and say a lot of things like hi grandma
I'm playing Superman so here the message
searches a three year old and the three
year old let's say knows about 500 words
in English and so we can label those
word one word two and through word 500
and each of those has its own
probability of being said by the
three-year-old we don't know those
probabilities but let's just say we've
measured his speech we've recorded lots
of hours of his speech and we can assign
these words different probabilities well
grandma doesn't know exactly what words
going to come next out of his mouth so
she has more surprise this time than for
his brother therefore this message
source has more information content it's
greater than zero bits now just to test
your intuitive understanding of this
idea let's have a very short quiz our
quiz has two questions which has higher
Shannon information content and it asks
you to compare two possible message
sources in question one and in question
two