The next thing I want to do is talk a bit
about the history of AI.
And, obviously, the history of AI
is going to be necessarily abbreviated and simplified
here, but I just want to give you
appreciation for how multifaceted the history is
and how rich and somewhat sometimes controversial it is.
So a natural starting point to talk about the history of AI
is Alan Turing's famous paper in 1950 called Computing Machinery
and Intelligence.
So in this paper, he asked the question, can machines think?
And he proposes the Imitation Game
as his solution, more popularly known as the Turing Test.
And some of you probably know the Turing Test
is said to be passed by a machine
if it can fool a human judge into thinking that it
is actually a human being.
So this paper is remarkable not because it
built a system or proposed new methods,
but it framed the philosophical discussions
of what is intelligence for years to come.
And you just have to appreciate how difficult a notion
intelligence is to pin down, so this was really
the first actionable formal answer to the question,
can machines think?
And now, whether you think that working on a Turing Test
is a good idea that will lead to progress
is questionable and controversial,
but at least philosophically it's quite thought-provoking.
So for us, one major takeaway of the Turing Test,
which was not really highlighted,
is this objective specification.
So note that the test itself is meant
to be capturing what a system ought to be doing independent
of how you get there.
It doesn't say whether it should be
using neural networks or logic based methods or so on.
And this modularity is going to be really important to us
in this course.
So at the end of the paper, Turing
does speculate on what might work.
So he talks about two possible approaches.
You could take a top down approach
and try to tackle abstract problems such as chess.
This is the route taken by symbolic AI.
You could also, quote unquote, "provide a machine
with the best sense organs" aka sensors
and teach it like a child.
And this is more of the approach taken by neural and statistical
AI, and both have been tried, and we'll
see how all three types of AI, symbolic, neural, statistical,
kind of meld together at the end.
So to start our first story, let's go to the summer of 1956.
The place was Dartmouth College.
John McCarthy, who actually founded the Stanford AI lab,
organized a workshop.
He gathered the brightest minds of the time in attendance
with Marvin Minsky, Allen Newell, Herbert Simon,
all of whom went to make seminal contributions to AI,
and the participants set out a not so modest proposal.
It was to-- they claimed that every aspect of learning
or any other feature of intelligence
can be so precisely described that a machine can
be made to simulate it.
So they were really after the moon.
They were after generality.
And this was post-war computers, which is coming on the scene.
It was a really exciting time, and people
were really ambitious.
So during this time, there are a few systems that were built.
Arthur Samuel built a computer program
that could play checkers at a reasonable amateur level
and actually featured some machine learning.
Allen Newell and Herbert Simon came up
with the logic theorists that could prove theorems.
For one theorem, they actually found a proof that was better
than the human written proof, and they
tried to submit a paper on the result,
but the paper got rejected because the reviewers said
it was not a new theorem.
What the reviewers didn't realize,
that the third author was actually a computer program.
Later they generalized these ideas
to the General Problem Solver, which
was aimed at solving any problem,
provided it could be suitably encoded in logic.
And again, this carries forward the ambitious general
intelligence agenda.
And this was a time of high optimism,
with the leaders of the field, who are all
really impressive thinkers, predicting
AI would be solved in a matter of years.
But we know that they didn't get it solved in 10 years,
and there were some tasks such as machine translation which
are very stubborn.
So there is now a folklore story,
I don't know how true it is, but it's amusing nonetheless.
You take a sentence like, "The spirit is willing
but the flesh is weak."
Translate it into Russian, which was the favored language
for translation in the '50s, and you translate it back,
and then you get, "The vodka is good but the meat is rotten."
So this was less than amusing to the government funding
agencies, who decided to write a report showing how really
machine translation wasn't going anywhere and cut off funding.
This led to the first AI winter.
So what went wrong here?
So there's two things.
First is that most of the approaches
involved casting problems as logical reasoning, which
required a search over an exponentially large state
space, and the hardware at the time
was just simply too limited.
And secondly, even if the research had infinite compute,
they would still not be able to solve AI,
because there's just too many concepts in the world.
Words, objects, and all this information
has to somehow be put into the AI system.
So these grand ambitions weren't realized,
but nonetheless there were some useful contributions,
many due to John McCarthy, that came out of this era.
First, Lisp was invented for AI, and arguably it's
still the world's most advanced programming language.
Garbage collection is something that, if you're
programming only in Python, it allows you to not know
what garbage collection is.
And time sharing, the ability to use a single computer
by multiple people was prescient at the time.
So then fast forward to the '70s and '80s,
knowledge was the key word.
And AI researchers thought knowledge was the key
to combat both the computation and information limitations
of the previous era.
And at that time, expert systems became very fashionable,
where a domain expert could encode
knowledge in the form of rules, usually looking like this.
And there is a noticeable shift as well.
The solve-it-all optimism from the '50s and '60s was gone,
and instead researchers focused on very practical systems
targeted at particular domains.
For example, chemistry, medical diagnosis,
and business operations.
And there were some good things.
Knowledge did help curb both the information complexity
and also restricted the state space so that it alleviated
the computational burden.
And this was the first time that AI
had real applications on industry,
but there were obviously problems.
Deterministic rules couldn't handle the complexity
and uncertainty in the real world,
and moreover, these rules just became quickly too
complex to create and maintain.
So this is a quote from Terry Winograd, who some of you
know was on the HCI faculty at Stanford.
But before he was HCI faculty, he
worked at MIT as an AI researcher,
and this is what he had to say in the mid '70s.
He thought that it was a dead end.
There were just too many complex interactions
between all the components, no easy footholds,
and you just couldn't have a mental model of what
was going on in your head.
And, moreover, there was a lot of overpromising
and under-delivering.
The field collapsed again, and it really
seemed that history was repeating itself.
So at this point, we're going to leave aside
the story of symbolic AI, which dominated
AI for multiple decades, and go back in time
to 1943 to tell the story of neural AI.
So 1943 is the year often attributed to the birth
of artificial neural networks.
So McCulloch and Pitts devised a simple model
and studied mathematical properties of the simple model.
But they didn't do anything in a way of learning the models
or parameters.
In 1946, there was the first learning rule
by Donald Hebb based on the mantra
that cells that fire together wire together.
It was nice and simple, but it didn't really work.
1958, Rosenblatt came up with a perceptron algorithm
for learning single layer artificial neural networks, AKA
linear classifiers, which actually turned out
to work really well and was used even fairly recently.
'59, there was an analog for linear regression
by Widrow and Hoff.
They came up with actually a multilayer generalization
called ADALINE, which was actually
used to eliminate echoes on phone lines at the time,
and this was one of the first real-world applications
of neural networks.
And then 1969, this was a big year.
So Marvin Minsky, Seymour Papert wrote
a small book called Perceptrons, and they analyze perceptions
with very mathematical properties,
and they had a little almost trivial result that
showed a that single layer perceptron couldn't recognize
the XOR function, and even though that is said nothing
about the capabilities of deep networks,
somehow this book is largely credited with shutting down
neural networks research and the continued rise of symbolic AI.
It's a really kind of interesting piece of history
and I encourage you to go examine it.
In the '80s, neural networks started coming back again.
1980 was the first convolutional neural network,
which was trained in a kind of an ad hoc way.
1986, Rumelhardt, Hinton, Williams
reinvented and popularized backpropagation
from multi-layer networks, and now training
became a little bit more principled.
1989, Yann LeCun devised a convolutional network that
was able to recognize handwritten digits
and was actually deployed for the USPS
to recognize zip codes, and this was
one of the first major success stories
of using neural networks.
But until the mid 2000s, neural networks research was still
fairly niche, I would say, and they were very notoriously
hard to train.
In 2006, this kind of started changing.
Jeff Hinton and his colleagues had
a paper showing how you could use unsupervised layerwise
pre-training to mitigate some of these effects,
and the term deep learning started getting
used around this time as well.
But it was really 2012, I would say,
that that was kind of a major break for neural networks.
So Alex Krizhevsky, Ilya Sutskever, and Jeff Hinton
wrote this landmark paper, which came up
with what is now called AlexNet, a convolutional network which
had huge, huge gains in object recognition.
And at the time, the computer vision community
was very skeptical, and almost overnight it completely
transformed the field.
Think about computer vision without neural networks today.
It almost feels like kind of a distant memory almost.
2016 was another big event.
AlphaGo defeated Lee Sedol in Go, something
that experts thought was still many decades away,
and that just kind of firmly more
established deep learning as a dominant paradigm in AI.
And this kind of continues even to the modern day.
But let's reflect so far.
So we have seen two intellectual traditions.
Symbolic AI, which has roots in logic,
and neural AI with its roots in neuroscience.
The two have fought fiercely over the decades
over philosophical differences, but I
want to suggest some food for thought.
Maybe there are deeper connections here.
So remember that in that McCulloch and Pitts
paper that introduced neural networks and arguably
the roots of deep learning?
Well, they spent most of the time
talking about how it can actually
encode logical operations.
And the game of Go, which is actually
a perfectly logical game designed
by a few elegant simple rules, but AlphaGo
used the power of pattern matching capabilities
of neural networks to solve this otherwise logical game.
So there may be room for more symbiosis than we think.
So now there's a third and final story
that you must tell to complete the picture.
So this story is not really about AI per se,
but it's about the influx of certain other ideas
from other areas that have helped shape and form
a mathematical foundation for AI,
and we call this statistical AI.
So machine learning is very popular,
but the idea of fitting models from data,
which is at the core of machine learning,
goes far back, even to Gauss and Legendre
at the beginning of the 19th century, who developed least
squares for linear regression.
Classification was also very early in statistics,
and AI also consists of sequential decision making
problems.
For deterministic versions, Dijkstra's algorithm
from the algorithms community for models with uncertainty.
From control theory, Bellman created Markov decision
processes.
And notice that all of these developments
largely predated the '50s and '40s, where AI really
kind of started springing up.
So you might have noticed, if you're paying close attention,
that where we left symbolic AI was at the end of the '80s,
but where neural AI started really gaining traction
was the 2010s.
So what was going on in between?
And what was going on between was
that there was a period where the term
AI wasn't really used, at least not to the extent
that it is today.
And I think that part of it was to add distance
to the failed attempts of the recent kind of AI winter,
and also because the goals were just more down to Earth.
People talked about machine learning,
and during that period there were two paradigms.
There is Bayesian networks developed in the '80s
by Judea Pearl, which provided a reasoning under uncertainty
framework, which is something that is symbolic AI didn't have
a satisfying answer for.
1995, support vector machines were
developed derived from ideas from learning theory
and optimization, and at that time, SVMs were easier to tune
than neural networks and really became the favorite tool
in machine learning before deep learning started taking off
again.
So to kind of wrap up, there's three stories
that we talked about.
Symbolic AI took a top down approach
and really failed to deliver on its original promise,
but it did offer a vision and built impressive artifacts
like question answering and dialogue system.
Imagine trying to do this on ancient hardware in the '60s.
Neural AI took a completely different approach,
proceeding bottom up, starting with simple perceptual tasks,
which the symbolic community wasn't
interested in at the time.
I compared machine translation with removing echoes
on phone lines, for example.
But in the end, it offered a class
of models and a way of thinking about data
that has proven capable of conquering today's
ambitious problems.
And finally, statistical AI foremost for us
will offer mathematical rigor and clarity.
For example, in the course, when we define objective functions
as separate from optimization or have a language
to talk about the complexity of a model in learning,
these ideas and language all stem from statistical AI,
and the course will actually be presented mostly
through the lens of statistical AI.
But I want to highlight all three views are
kind of compatible and just offer different advantages
on the same underlying ideas.
Stepping back, the modern world of AI
is kind of like New York City.
It's a melting pot that has drawn largely
from a lot of different fields, statistics, algorithms,
neuroscience, economics, and it's really
a symbiosis between all these fields
and how they come together and allow
you to tackle real world applications that
makes AI so rewarding.
OK, so that ends the AI history module.
You can read much more about it at a few links
at the end of these slides.