[music]
-I think what we can do with trackers,
centers, and other trace measures is exciting,
but at the same time, I think we have an unprecedented challenge
in communicating that these tools are fair and valid.
Ultimately, it doesn't matter what a tool is capable of doing if people
and organizations see it as harmful or invasive.
-I think the most fundamentally exciting assessment technology
is natural language processing.
It has broad applications
against many different specific assessment types
and the machine learning approaches associated
with it have reached a maturity level
where it might actually improve contract measurement
beyond what we can currently manage.
It has broad potential, and it can be feasibly implemented
right now with existing technology,
and that really makes it both exciting
for research and practice.
Longer-term, there are a lot of other exciting possibilities
that rely on further advances in AI, like mixed reality or biometrics.
I think we're at least 10 years away from those
even to potentially become mainstream.
-To me, the most exciting workplace testing
and assessment technologies
are actually the ones that ultimately are most useful
and effective to job seekers,
employees, and organizations.
This seems like an obvious thing to be excited about,
but this point offers a challenge,
and that is to define what actually is useful and effective.
What does that mean?
There are certainly new technologies
that seem exciting such as measuring job applicant characteristics
or job performance criteria, using new technologies,
big data with multiple modalities,
and measuring intensely across time.
How do we know that doing so,
measuring in that way in those exciting technological ways
are actually useful and effective?
[music]
-I think there will be a number of anti-assessment tools marketed
to individual job seekers.
I'm imagining something like a deepfake or GPT-3
that is designed to trick the algorithms, and as a result,
assessment technology will have to evolve
to stay one step ahead of these tools.
How can you design a natural language processing personality assessment,
for example, that can differentiate between real and artificial language?
-I think the next big area for workplace testing
and assessment is computational psychometrics.
It's an interdisciplinary field that combines psychometrics
and computer science.
It's really about finding ways to map psychometric concepts onto the data
the new technologies provide while still maintaining meaningful
construct measurement.
It's interdisciplinary, that's a big challenge,
and it requires authentic expertise
in both machine learning and in traditional psychometrics,
which is not a set of skills commonly combined,
at least right now.
-I would like to think that industrial-organizational psychology
is going to become even more important in the era of AI and machine learning.
In other words,
I don't think IO psychologists are going to be replaced
by these modern technologies and algorithms.
Instead, our expertise on the science of work
in the workforce
is going to provide the necessary content,
the necessary expertise that's required to collect data effectively,
understand these technologies,
and how that combines with the data
and the algorithms to lead to effective decision making.
[music]
-Many people don't understand what assessment is,
and they believe it to be inherently unfair
because it chooses winners and losers.
At the same time,
there's this whole slew of new technological development
that is advancing without any consideration
of fairness or validity.
The biggest challenge we have
is to help people understand the difference
between an assessment that has been developed according
to the standards
and one that has not,
so they're not lumping all these things together.
-I think the most critical issue the assessment community needs
to address is this uncritical acceptance of classical psychometric methods.
There are a lot of people with overly simplistic views
of some psychometric concepts.
Say, for example,
to assess reliability you must calculate a coefficient alpha
or a test-retest coefficient,
but that forgets the actual purpose
of reliability estimates is to estimate the proportion
of true score variance out of observed, no specific statistic is required
to get to that number.
Another example is validation
where people often assume that finding a high correlation
with a self-report measure is the only way to prove
that you're measuring something meaningful,
yet if we could achieve the measurement quality
we want with self-report, why go to all the effort to learn
and integrate new technologies in the first place?
A high convergence correlation,
just like a high coefficient alpha, isn't really the goal.
The real goal is measurement.
-There's always been a concern about measuring job applicants
on job-relevant characteristics.
That's always been a long-standing concern in IO psychology,
and we conduct job analysis
to make sure our measures are well aligned
with the constructs, the characteristics we think we need in our job applicants,
that's a centuries-long pursuit.
The influx of AI machine learning technologies
does not make that practice irrelevant.
I guess what I'm saying in answering this question is,
in fact, it's more relevant or the concern for substance behind AI
and algorithms is actually growing
and that's where I feel the field needs
to be heading to increase its impact
and relevance within the domain of personal selection and AI.
[music]
-I understand the frustration of feeling that a test can determine your destiny.
Ultimately though, evaluating people is necessary
whenever there are fewer opportunities
than people who want those opportunities.
Most other ways of choosing people have serious issues
with bias and unfairness.
Assessments, when created properly, are a way of evening the playing field
and giving people with talent an opportunity to be noticed.
This is most noticeable
when it comes to issues of social and economic inequality.
People who need to work full-time, for example,
don't have the luxury of signing up for unpaid internships
or doing lots of extracurricular activities,
and they would be overlooked if selection committees
unduly weighted those activities.
-A lot of the data scientists that I've talked to seem to believe
that unfairness is something that can be engineered out of a system.
They often don't recognize or even think about the systemic issues
that create the problems that they're trying to engineer out.
In that way, mechanical solutions to equity issues
are sometimes more harmful than helpful.
They can actually create more problems than they solve.
Artificial intelligence by itself solves no problems.
It's all about how we use it.
It's only by combining our expertise across disciplinary lines,
by having open and frank conversations about these issues
that we're really going to ever achieve any real progress
in creating fair assessments.
-Regarding diversity,
equity and inclusion,
there's what I call a happy coincidence
that I think is playing out here in the arena of AI
and machine learning.
The coincidence has to do with the fact
that big data techniques
at least has the promise,
if the reality is working toward this promise,
of reliably measuring a greater diversity of job-relevant applicant characteristics.
We can measure many more things than we could before.
We can measure it over time,
we can measure it quickly, we can measure it relatively,
conveniently, and across behavioral
and self-report channels we can get
at convergent and unique aspects of characteristics.
[music]