[MUSIC]
I've been thinking about how technology's affecting the economy and
the workforce for a long time.
And there are lots of things I can talk about,
but let me pick up a couple of themes.
One is that, while we have seen some productivity gains since the 90s and
continuing, one thing that's been a little disappointing is that the workforce
hasn't been sharing that as much as we would have liked.
Median wages have kind of stagnated since the 90s,
even though productivity is at an all-time high.
And part of that has to do with, we haven't reorganized work to take advantage
of the technologies as well, and people aren't sharing in that.
What I want to zoom in on today in particular is one really amazing
technology, machine learning, which I see as one of the most
powerful general purpose technologies for us ever to experience.
And we're still in the early days, but we've been doing some work with my team to
try to understand better what the workforce implications are.
So I'll zoom in a little bit on that, and what its implications are for wages and
for restructuring of work.
So to preview the results, in case you zone out at some point,
you can just concentrate for the next minute really carefully here.
Many applications of machine learning have achieved even human level or
superhuman level.
I'll show you some data on that, but you've seen the examples of that.
But at the same time, we're very far from artificial general intelligence,
like these robots that I have on my front slide that can do the the full range of
things that humans can do that Hollywood features.
So that raises a natural question, at least to me, which is, what are the things
that machine learning can do well, and what are the things it can't do well?
Can we separate out those categories?
It turns out there’s a very distinct set of things that machine learning is
superhuman at, and another,
much broader set of things that machine learning is useless for.
And what we've done is, we've talked to a lot of experts about classifying these.
And we developed what we call a suitability for
machine learning rubric, working mainly with Tom Mitchell,
who's head of machine learning at Carnegie Mellon.
But interviewing a whole bunch of people,
Fey Fey here at Stanford, Daniela Ruse at MIT.
Most of the people were willing to give us a bit of advice on our rubric.
And over time,
we evolved it to come up with a pretty good set of criteria, we think.
And then we apply that rubric to all the tasks in the economy.
It turns out, leading off,
there's a list of all the tasks in the economy that the workforce does.
Not sure it's completely comprehensive, I'm sure it's not.
But O*NET categorizes the economy into about 964 occupations.
And for each of those occupations, it breaks it down into the tasks.
These are the things that a bus driver does, this is what an economist does,
this is what a kindergartener teacher does, a radiologist, a lawyer, etc.
Typically, they say there's about 20 to 30 tasks for most occupations.
So we went to every one of those tasks and
tried to see, is that particular task suitable for machine learning?
So we didn't try to do it at the occupational level,
we don't think that's the right level.
But at the task level, we have a better chance of making some progress.
And there's a long literature in economics about this task level approach by and
David Otter and others.
And so the key results we came away with were that the set of tasks that
are affected by machine learning are fairly different from the ones that were
affected by earlier waves of automation.
The stuff David Otter and others have described that were routine computer
automation affected over the past, say, 20 to 30 years.
In particular,
there are now a lot of high-wage jobs that are significantly affected.
And you may have heard financial analyst, parts of the legal profession,
radiologists, and many others.
There are tasks that they do that machine learning can now do quite well.
However, it's also important, maybe even more important, to note that we didn't
find any occupation where machine learning just kind of ran the table, and
was able to do all the tasks in that occupation.
Think about, if an occupation has 20 or 30 tasks, it's not surprising that machine
learning can do some of them, but not every single one of them.
Now, to be sure, there were some occupations where
it covered a big chunk of them, and others where didn't make as big a dent.
But the general takeaway was, most occupations had some tasks that were
suitable for machine learning, and many other tasks that were not suitable.
So what does that lead you to take away?
It's that it's not going to be so much a matter of taking entire occupations and
just making them disappear.
It's more a matter of having to reengineer and redesign things.
Parts of the job can now be done better by machines, parts of it can't.
So how do we reorganize work to unlock that potential?
That's a very hard thing to do, is to reorganize and
reengineer the way processes work, and the way occupations are organized.
Also, another takeaway I won't talk as much about today is that the more
measurable tasks tended to be the ones that were first affected by machine
learning.
Those of you who work in the area, you can see why that would be true,
especially reinforcement learning, but also supervised learning systems.
That has some interesting implications about the kinds of incentive systems that
might emerge.
What kinds of things will be left for
humans to do if the more measurable work is increasingly done by machines.
You can think about that a little bit in terms of principal agent problems or
incomplete contracts.
And we also could roll these up in different ways.
You could aggregate the tasks at the occupation level, as I've been suggesting.
But you can also aggregate them up to firms.
We have data from places like LinkedIn, and you can see that some firms,
General Motors does very different tasks on average than Walmart does, or
than JP Morgan does.
So different firms themselves have different vulnerability scores,
different sets of likelihood of being affected.
You can also roll it up to geographic regions.
The work that's done in Kansas City is different on average than what's done in
Manhattan, or that's done in other parts of the country.
There are some similarities, but also some differences.
And of course, industries vary as well.
And so if I had to leave you with one bottom line,
it would be that there's going to be a lot of redesign
needed to get the full productivity benefits from machine learning.
So the biggest workforce implication is not what I think a lot of people
are talking about, this big chunks just being automated or mass unemployment.
But rather, aa need for people, HR people, managers,
entrepreneurs, to be more creative about how we reinvent work in the workforce.
So let me give you some more data, fleshing out each of those points.
As I think almost everyone in this room knows,
machine learning has gotten much better.
A very fundamental capability of humans is vision, recognizing objects.
And one way to measure that we have at the AI Index project, I encourage you to go to
this website for lots more data like this, is the progress on image net.
And you can see that the blue line there grows very steeply.
Geoff Hinton and
his colleagues in particular in 2012 introduced deep learning techniques.
And suddenly, we were able to solve problems much,
much better by some measures.
They can recognize different breeds of dog, and different types of jaguars and
leopards better than I can, and better than most humans can.
So it's gotten to superhuman, even, performance in some of those.
I don't want to say that vision in general is anywhere close to being solved in
the wild, it's still issues.
But on different data sets that we focus on, we've done very well.
You get similar improvements in other areas, you can test for yourself.
Siri, not perfect, but we're in this interesting, say,
10 or 15-year period where we went from not being able to talk to our machines
to routinely talking to machines.
If you you see somebody talking on their phone in the courtyard out there,
you don't really know, are they talking to another human, or
are they talking to a machine, it could be either one.
And of course, that leads to many specific applications.
Hardly a month goes by that we don't see some article on archive or nature or
science of a machine learning system outperforming a radiologist or
a dermatologist or a pathologist on some task.
And just early days, but beginning to see some economic impact, not a whole lot yet.
One place we saw it, I just had a paper come out last month looking at machine
translation, and how that has dramatically improved the sales of eBay for the pairs
of countries where they were able to have the same language pair, by 11% more sales.
Which is equivalent in a gravity model to making the world 20% smaller,
just from that one application.
And there are many other applications, everyone in this room has a set of
applications like these that you can implement.
They all have this common pattern of mapping a set of inputs, x,
to a set of outputs, y.
It's a general concept, but
there are many specific applications of it where it works very, very well.
And most of them have not yet been exploited yet.
So what does this mean for the workforce and the economy?
Well, the first thing that comes to mind is substitution effects.
And that's the one that I think of first, most people think of first.
Is that if a machine can do it instead of a human and machine is cheaper,
then entrepreneurs, managers are going to substitute the machine solution for
the human solution.
That's certainly one important one, but if you think a little bit more about it,
you realize there are many other economic implications.
Sometimes machines are complements, not substitutes, and
they actually increase the value of labor.
In fact, not just sometimes, through most of human history, at least the past couple
hundred years, what's happened to the price of an hour of labor?
It's gone up, so mostly, our technology has been a complement to an hour of labor,
not a substitute to an hour of labor.
Demand elasticity, as the price goes down,
you actually may spend more, depending on what the price elasticity is.
Seems a little paradoxical, but if you're an economist,
you know that happens all the time.
Income elasticity, supply elasticity, I won't go through all of them.
The last one I want to underscore, though, is this reengineering is another
possibility, and so it's described more in this article.
So what we did, as I mentioned, was we applied this rubric to data initially from
O*NET, but we're now doing other data sets like LinkedIn, Burning Glass.
And we had a lot of different people score it, so we got some reliability,
couple of articles wrote this up.
Here's some of the questions that came out,
like is the task information recorded by a computer?
If it is, that makes it more likely that you can use machine learning,
you need data.
Is a task repeated frequently?
Can it be completed in one second or less?
Some of you may know, this is Andrew rule of thumb for whether or not to assign
people to try to build a machine learning system, and there are many others.
And here's some of the tasks we applied it to.
People like to talk about radiologists, so let's look at that one.
I knew that they interpreted medical images, that's an important one.
But they do all these other things, like sometimes you have to conduct a physical
exam to inform decisions about appropriate procedures.
Machine learning is great for medical imaging, not so
great at doing physical exams or many of the other tasks there.
So this shows you that there are going to be parts of the job that are suitable and
parts that aren't.
Now you can roll up for
every single occupation to what extent the average score is.
Each of these are weighted, by the way, by how important they are,
some are more important.
Medical images is a particularly important thing for radiologists.
And then you get a scatter plot that looks a little bit like this,
there's clearly a pattern there.
If you look at the low-wage jobs on the left to the high-wage jobs on the right,
on average, low-wage jobs are more likely to be affected.
Like cashiers, many of them already have been automated.
But you get some high-wage jobs, right in the middle is title examiners.
You get some high-wage jobs like airline pilots,
they're among the highest paid professions.
They also are increasingly automatable by a lot of these technologies.
And of course, I couldn't resist checking economists.
>> [LAUGH] >> Not paid as much as they should be,
clearly, but not super machine automatable, at least not yet.
You can roll this up to the level of the economy,
and you get about $700 billion worth of tasks in the economy
that are very high on both machine learning and on measurability.
Depends on where you draw those thresholds, but we drew them pretty high.
You can also, as I mentioned, roll it up to different geographies.
I won't go through, but we have lots of colorful maps of machine
learning in general, or prediction, or image recognition.
There's some jobs that will have a lot of image recognition as part of their,
text data, speech data, etc.
And you get different patterns that are likely to be affected there,
overall structured data use, deep learning.
So you can slice it different ways, and make predictions about which
regions are likely to have a lot of jobs that are affected.
You can do it by the industry level, the hotels and
transportation, retail trade tend to score pretty high.
You can do it for firms, this one, there wasn't much of a pattern,
at least not yet.
I might predict, and we'll see, we can check back in a year or two, whether or
not the slope of that line is still zero.
But right now, those companies that have more tasks that are suitable for machine
learning are not significantly different in terms of adjusted market value.
And you can group them all sorts of other fun ways.
In some of our papers you can see, for instance, you can create these networks of
occupations that share a lot of tasks in common, for instance, clerical workers.
Whether they're in the healthcare industry or the retail industry or manufacturing or
finance, they do a lot of the same kinds of tasks, as do factory workers.
And they are particularly vulnerable to machine learning.
Other jobs like scientists, whatever industry they're in,
are somewhat less vulnerable, so are therapists and clergy,
so if you want to be relatively safe for now.
And then there are many other clumps that emerge as you look at
the clusters of jobs that are related to each other or similar.
So to wrap up, I'd say that there are different kinds
of jobs that are going to be important in the machine learning economy.
Some of those ones I just described, there will be a lot of substitution and
replacement of at least parts of them.
But I think underrated is the importance of redesigning and
reinventing how work is done.
Bundle, take a set of tasks that can be done by machine learning,
separate them out, bundle them for a machine to do, design a new occupation.
Maybe if you take radiologists,
maybe there's still an important set of work to be done there by humans.
But part of it the machine will now do part of it, and the human,
you want to have a division of labor there.
Being able to measure things more precisely becomes a lot more valuable if
machine learning is supercharged on those measurable tasks, so
we should think about how to do that.
And just more broadly, how to innovate by learning to ask better questions, and
where to apply machine learning to different kinds of tasks.
So to conclude, we found that machine learning is indeed human or
superhuman in some tasks, but not all tasks.
It's a broad, general purpose technology that's affecting many industries,
many occupations, many regions.
But you can separate out where it's likely to be more effective and
where it's going to be less effective.
We do that with this rubric, and as I mentioned, you can map it onto geography,
firms, industry, clusters, lots of other ways.
And I think the main takeaway I'd like to leave you with is not to
think of just mass substitution or automation, but
rather a reorganization of work as being the key driver.
[MUSIC]