One of the reasons that we wrote this book is to teach people to be skeptical.
So there is an old book called How to Lie with Statistics, which teaches people to catch
all of the tricks, like four out of five doctors recommend.
Well, what exactly was the sample of doctors that made this recommendation?
Was it literally just five, and four of them were the cousin of the people that-- the toothbrush
that's being recommended?
So we wanted to write something similar for AI.
And the reason for that is because AI is hyped all the time.
A lot of journalists write by a press release, essentially.
What happens is, Microsoft puts out a press release.
Or some other company.
I don't mean to pick on Microsoft.
But I'm thinking of a specific case.
Puts out a press release that says, hey, we've got a AI system that's reading as well as
human beings.
And then the press just kind of accepts that often.
They report it.
And in fact, they embellish it.
So now it's not just, they read as well as human beings, but machines are superhuman
in their ability to read.
And then they'll throw in, oh, and millions of jobs are at stake.
Like, this actually happened.
And then you go and read the study, and it just means, on one little narrow aspect of
reading the machine is maybe equal to people or better than people, or it's a little bit
better than last week's.
It's not really that exciting when you actually go and read.
It's nowhere near actually reading.
So it can underline parts of a text, but it's not the same thing as being able to read between
the lines, which is really most of what we do.
Reading would be really boring if the author spelled out absolutely everything.
Instead they kind of let you play at home and come to some conclusions yourself.
Well, we don't have AI that can do that at all.
But you see this thing in the newspaper or some website saying, you know, machines now
are superhuman readers.
So what we offer our readers are six questions that they can ask every time they read anything
about AI in the news.
So the first thing you should ask is, is there a demo?
Can I try it out for myself?
If there's no demo and you can't find out for yourself, there's a reason for that.
It's because it's not really stable yet.
So they've got some version of this that works in some limited context, but it doesn't mean
that it really works for real.
So that's one of the questions.
You ask, is there a demo of this?
Another question you ask is, how general is it?
So, OK, deep mind has won at AlphaGo on a 19 by 19 board.
Is there any evidence if you trained it on a 19 by 19 board that it would then be able
to transfer what it learned to a board that was 9 by 9, or 31 by 31, or wasn't a squared
all?
A human go player can play on a board of any size, you know, if they're expert enough.
But that particular system would have to start all over.
It was trained on, I don't know, 10 million games.
It would have to take another 10 million games in order to be able to play now on a board
of a different size.
So it doesn't generalize what it learned.
And this is often the case.
another thing to do is to strip away the rhetoric.
So somebody says, this system is reading.
Well, is it really reading?
Or is it doing some narrower task than that?
Well, maybe you claim that it's reading.
But if you think about all the things that go into reading, that includes inferring things
that you don't already know, building some kind of internal mental representation of
what's going on, who did what to whom, and so forth.
Does the system that, quote, "does reading," does it really do that?
Or is it just underlining text that's relevant to a passage that you saw?
So if you read one of Aesop's Fables, you can explain what the moral is at the end of
it.
Can this system do that?
Or can it just underline the place where the animal packed away food for the winter.
It'd have no idea why it did that.
Going back to How to Lie with Statistics, suppose somebody tells you, this machine works
better than humans.
You really ought to ask, which humans?
And most of the time the humans are people on Amazon Turk who get paid $0.02 per item
and do 300 items in order to make $6.
And they're bored out of their mind.
They're probably not human experts.
Unless we're talking about a game like go where machines get pitted against the world
champions, you're probably talking about average humans that are bored out of their mind as
the control group.
That doesn't really tell you what best human performance is.
And you really ought to check when you read one of these studies.
Another thing to watch out for is, OK, we've made progress on this particular task.
But is this really a step towards general AI?
So if you have a system that can tag photos pretty well, that's exciting.
And it's useful.
Who wants to sit there labeling all of their photos?
But does it mean just because you can tag photos that you have a system that's giving
you general intelligence?
Or is it just, well, you know, when you tag the Eiffel Tower, a lot of other people did,
too.
And so the statistics of the Eiffel Tower and the images make it pretty easy, if you
have a big data set like Google does, to tag that photo.
Tagging the photo might just rely on something fairly superficial, which is like comparing
similarity to a large database.
That doesn't mean that your system actually understands what a tower is, what Paris is,
why somebody would climb it or think it's beautiful.
And so it's possible to make little, narrow slices of intelligence that seem more intelligent
than they really are.
And you go all the way back to the history of AI, and there was this system called "ELIZA,"
which just did basically keyword search and had psychiatric replies.
So you say, I'm upset with my mother.
And it says, tell me more about your family.
It's just looking for the word "mother" and saying, tell me more about your family.
And people were fooled into thinking that it's deep.
That it's-- some people thought it was real, and other people thought it was a major advance
in artificial intelligence.
Instead it was almost built as a prank.
The truth is it wasn't that smart.
The chat bots we have today aren't a whole lot smarter.
So just because you can have a five-minute conversation with a machine doesn't mean that
it's really an advance towards intelligence.
It might be just kind of a bag of tricks.
The last thing is, you want to know how robust something is.
So maybe, for example, you have a system that will work for blind people to label the images
that it sees.
And it'll tell you there's a camera over here, there's a bookshelf there.
But you'd really like to know, does it work in the dark?
Or does it work in strong sunlight?
You want to know how robust this is.
I actually built my new company around this idea.
It's called "Robust.AI."
And the goal of the company is to try to take a new approach to AI where things work in
general rather than just some laboratory demonstration.