John Sumser: We did 450 hours of interviews and demos
with 110 companies to get the core data. And then
we ran quality, a quantitative survey across the
top of that. So it's a rich data set. And what we
found is that we thought that the number of
intelligent tools or AI providers was going to
grow at a certain rate over time. And instead,
what happened in the last year is AI became
ubiquitous. And so in your organization today,
there's all sorts of AI already and it's in
Microsoft Office, and it's in Google and it's in
all of the browser-based tools that your employees
use every day. It's also in every single piece of
HR software. That's a development and so you may
not have it in every piece of HR software that
you've got, but it's coming down the pipe. And
every one of the people who supply you this stuff
has an AI project and they're working on embedding
machine learning in every possible aspect of the
tool.
Another way of saying that is that we've gotten
smarter.
So one of the things that we have known all along
is that the narrower the problem, the better the
result with this stuff. So what you need to get a
really good result is a well defined problem and a
rich data set underneath it, and then you can do
really interesting things with it. And what's been
happening is the definition has been growing out
like this. So I would say there's sort of three
developmental things that have happened to
providers of this stuff. One is companies that had
a simple function. They were a chatbot company or
they were recoupment process company of some kind.
Some of them have gotten deeper, and they've
gotten deeper and smarter about the single problem
that they were trying to solve. Some of them have
gotten broader. So they've added major
functionality in a sort of a horizontal way. And
some of them have fled the marketplace. Because
there are there is one of the things we cover in
the new report is there's an extraordinary failure
rate of this stuff. If you're talking about
chatbots, something like 70% of all chatbot
implementations get turned off. They get turned
off not because the chatbot doesn't work, but the
business of managing a chatbot includes all sorts
of other implications like changing the jobs of
the people who do the stuff that used to be done
via the chatbot or the budgeting associated with
maintenance. And so it becomes a difficult thing
to manage. Cause it didn't look like it would be a
complicated process, when it started it looked
like it would simplify things and it turns out
that getting this stuff in is complicated. And the
chatbot producers have gotten smarter about how
they do implementation. So while there's a big
failure rate now, it's early, and the failure rate
will continue to decline over time, and the
technology will get better. So the answer is sort
of yes, the problems are different and the
understanding of what it takes to actually solve
the problem is evolving as well. One of the things
that's in the report is our annual watch list and
the annual watch list has a dozen vendors whose AI
products represent the edges of what's possible in
AI. The panel today had four of those vendors.
They chose the vendors for the panel based on the
fact that there was no observable connection to
what they did. So imagine that there's a matrix
and one -- the vertical axis of the matrix is
point solution at the top suite at the bottom --
and the horizontal axis is data management at one
end and data monitoring at the other end. The
players were swoop talent in the data management
end. KeenCorp in the data monitoring end, ultimate
software in the sweet end, and modern hire in the
point solution end. Modern hire does job
simulations and by job simulations that means that
they give you the experience of what it feels like
to do the job and then assess whether or not
you'll be good at the job by having the experience
of doing the job. And they've recently integrated
with a video interviewing company. So there's this
one sort of, and analytical tool that moves people
from the interview stage to the job simulation
stage and takes data all the way through and
learns about what works and what doesn't work. At
the bottom of that axis is ultimate software. And
ultimate software is such an interesting story.
They are many years ago, they were a payroll
company, they became a human capital management
company. And so they've got a full suite of human
capital management capability. And about three
years ago, it's like they bought an old house and
lifted it up off its foundation so they could put
it in a new foundation and the underlying
foundation is AI from a company that they bought
called Cam Joya and Cam Joya looks at employee
sentiment as expressed in regularly scheduled
questions that are sent out to the workforce. And
they get this very clear picture of what's going
on inside of the workplace across all of the human
capital management functions. On the horizontal
axis, you've got SwoopTalent, and SwoopTalent is
really interesting. There's a joke about machine
learning that is that goes like this: Machine
learning is 80% data cleaning, and 20% complaining
about data cleaning. And what SwoopTalent does is
it automates a bunch of the process associated
with data cleaning. The biggest problem in data
cleaning, is getting fields that have different
names to match up so they go together so you can
do a big transform, and their tool uses machine
learning to assess the actual content of those
columns of data, so it can tell where two things
would go together without having to have a pivot
on the top. At the far end of that spectrum is
Keen. And KeenCorp monitors tension in the
organization. And when they say tension, what they
mean is, the language that people use to talk
about work varies over time. And as you examine
the flow of digital communications, you can start
to see the language change based on what's going
on in the organization. And that can tell you
where embezzlement is happening or sexual
harassment is happening. And so you can use this
tool as a team to monitor the data flow and use
the data flow as a trouble indicator, and it
suggests a world in which instead of waiting for
the victim of sexual harassment, to do the
reporting, you can spot the propensity for sexual
harassment in advance, and like a spill on the
shop floor, go and clean it up. Alright, so it
turns things that have been reporting problems
into health and safety issues. So those are the
four that were on the panel. And each of the
players, there was either the CEO or the head data
scientist for each of these companies. And what
was really interesting about the panel is they --
because I put them together -- they started to
understand where they might work together, because
they were so different from each other, that they
represented gaps in the work. And so the panel
itself was was astonishing, because it ended with
these guys who had just done demos, now talking
about how they could use each other to improve
each other. And so it's a really wonderful and
powerful panel. There's a movie that I think you
download on YouTube called Those Magnificent Men
In Their Flying Machines. And it's comedy from the
60s, I think. And it's the story of the early days
of aviation. And what happened in the early days
of aviation is there were all these ideas about
what might work, there were 800 airplane companies
before the Wright brothers flew their plane. And
those 800 airplane companies, all but one of them
suffered crash after crash after crash after
crash. And then the Wright brothers flew. And it
was necessary for investors and customers to be
part of the ecosystem of all of those 800
companies in order to get the one that flew. And
what happened over time was like what happened on
the panel that I was talking about. The people who
were the mechanics in those innovation
laboratories of airplanes, talk to each other,
learn from each other about what worked and what
worked in spite of the fact that they were
competing with each other. And we're at that stage
now. The interesting thing about aviation is it
took 20 years to get from the idea to the first
flight. It took another 20 years before there were
factories producing horribly expensive planes that
would fly one person at a time around. And it took
another 20 years before Howard Hughes could invent
the airline system, which was -- it's the dumbest,
simplest idea, but nobody was able to have it. And
that is an airplane is a bus. But nobody could see
that because the original model was a man flying
and we're on that kind of trajectory where we
think we know what we're headed towards now. And
it's interesting, we are sort of full of these
processes that look like they can be automated,
but like in the case of recruiting, much of the
recruiting intelligence is being applied to a
problem that's got a 50% failure rate. And so
because there's a heavy emphasis on return on
investment that people talk about, the vendors are
saying, "Oh, you can do this faster and cheaper
with machine intelligence embedded in the tool."
And the question is, do you really want to do
something that fails 50% of the time, faster or
cheaper? And we're hitting those questions really
hard. And the only way that you can discover how
dumb they sound, is by going after them. You hear
a lot from critics who aren't really versed in
technology evolution over time, that there's
something wrong with the vendors, that there's
something unethical going on here or that there's
some set of claims that are untrue. But what's
really happening is, we're learning how to think
about this and in learning how to think about
this, you have to try a bunch of things on and
some of them fit and make sense and some of them
don't fit and make sense. And it's part of the
early evolution of a brand new technology that
we're going though. Technology is invented by
human beings and always contains the bias of the
inventor, it's not possible for human beings not
to have bias. And it's not possible for them to
invent things that don't have bias. Now, whether
the bias is desirable or not, is a separate kind
of question. And often, when people talk about
there being bias in AI, what they really mean is
illegal bias of the kind that discriminates
against people in protected classes. And it's
confusing because yeah, that's bias. But the
universal bias is a massive, massive environment,
and what we know -- there's this great experiment
at Amazon. Amazon tried 500 different data models
in the recruiting system trying to get the
predisposition of their hiring process to hire
males changed. But they always use the historical
data and what they found was that the data is
biased, right? And so this technology takes data
and turns it into rules. That's what it does. And
so whatever the history of the data is, is going
to be reflected in the rules that are output in
the thing. And that's, that's just, that's just
the thing. And so what that means, what that
really means is that when you get an output from a
machine, you can't ever believe that it's the
truth. It's the machine's opinion. And as the
consumer of that opinion, you have to take that
opinion and make a decision and the one thing that
you should never ever, ever do is let the machine
make the decision. Never, when it comes to people
and their livelihoods, and their access to
opportunity, machines don't have compassion or
common sense or intuition, or any of the things
that make really good decisions happen. They just
have an input. When you have AI -- this is worth
thinking about -- when you have AI, it's like you
just hired the best historian in the whole world.
And this historian has access to everything that
happened in the past related to this particular
problem. And what this historian is really good at
doing is telling you what would happen if nothing
changed from the past. But the reality of the
world that we live in is it's driven by emergent
phenomena, and so it's never the same as what
happened in the past. And so it's very useful to
know that on the first Sunday of September every
year for the last 10 years it rained. It's really
useful to know that it means on the first Sunday
in September, you probably want to take an
umbrella. But the machine isn't wrong if it
doesn't rain. You have to make the decision about
what to do with the thing and so when it comes to
liability, the machine gives you the input, but
you're the employer and you have the liability and
no court in the world is ever going to change
that. You make the decision, the human makes the
decision and so the machine holds no liability.