Transcriber: Phạm Linh Nhi Reviewer: Maria Pericleous
Good evening, Craig Watkins,
as was noted in the introduction,
a visiting professor here at MIT this year,
my home institution is the University of Texas at Austin,
So. Hello there.
Bob Metcalf from UT Austin, from the City of Austin,
and I’m also being hosted
by Fotini Christia and Fox Harrell.
Fotini is a member of the Political Science Department here,
but also the Institute for Data Systems and Society,
which is sort of my host department while I'm here this year.
You know, Fotini and I talked earlier this summer about me coming here,
and Fotini and some of her colleagues in IDSS
launched a new initiative about a year and a half ago,
designed to understand the relationship
between Artificial Intelligence and structural racism.
It was something we were beginning to do at the UT at Austin,
through a campus-wide initiative Grand Challenge called Good Systems,
where the University is invested in research
to bring researchers from different disciplines
to think about some of the big challenges,
many of which
you’re hearing about here tonight, in terms of AI machine learning
and its impact on the world around us.
Many of you probably know that MIT is launching
a new school of computing,
and one of the things they want to do as they build this school,
is to make it interdisciplinary,
and also to design it in ways
that can speak to these structural realities,
the systemic forms of inequality that are so important,
and so helping students who come through at MIT.
We’re doing the same thing at UT Austin
develop the kind of capacity
to develop the skills, the intellect, to leverage these systems,
leverage this technology in ways that have real world impact.
And in order to do that,
you have to understand what’s happening in the world in which we live.
So these are two images,
you may not know these two gentlemen,
but they were both in, one in 2020... I’m sorry.
Let me go back.
One in 2019, one in 2020,
were falsely arrested
for a crime that they didn’t commit,
and they were falsely arrested by facial recognition software,
the system in some ways erroneously identified them
as the culprit of a crime that had been committed.
They both were apprehended, detained
and spent some time in jail until the situation was clarified.
It’s circumstances like this
that raised the alarm around the world
about the importance of these technologies
and the ways in which, if we’re not careful,
they can replicate the systemic forms of inequality
that we oftentimes hear and talk about.
This was covered in the press and covered in the media,
and it created a really interesting conversation,
both within academia, within industry,
and increasingly, I think, around the world as well.
How do we respond to these systems?
How do we design these systems?
How do we rethink them, redesign them
in ways that mitigate these disparate impacts,
that mitigate the likelihood of replicating these fronts of inequality
at massive scale?
And so this is something that has become an important part
of a conversation amongst a number of different communities,
clusters of thought leaders and thinkers,
including machine learning practitioners.
And over the years, they have developed a particular sort of framework,
in which they think about this issue,
the notion of fairness,
and maybe you’ve heard about good AI or fair AI.
This is something
machine learning practitioners are trying to design,
something they’re thinking about theoretically,
and also thinking about the technical properties
of what it means to actually build systems,
to build computational systems that function and operate in fair ways.
Many of you here are academics, it probably comes as no surprise to you
that something seemingly as simple as the notion of fairness
is actually quite complex.
And so you have roughly 20 to 25 different definitions
and notions of fairness that have been developed,
or that are being developed by machine learning practitioners.
Just to give you a glimpse
of some of the kinds of conversations they’re having.
If you’re trying to design a model
and you want that model to be fair,
you want that model to treat likes alike and not discriminate,
not exercise bias,
in the case of race, for example,
do you design that model to be aware of race,
or do you design that model to be unaware of race?
Some would argue that, by designing the model so that it’s unaware of race,
you end up not recreating the kinds of racial biases
that we’ve become accustomed to, historically, and even contemporarily.
But others would argue the exact reverse.
And that is, if you don’t explicitly recognize race,
you end up recognizing race via proxy, right?
So if your model is looking at other kinds of data,
looking at other kinds of information,
zip code, school, your financial situation,
all of those things, in some ways,
can be a reflection of or a proxy of, one’s racial identity class perspective.
And so, in that sense, the idea is
that even when you’re not explicitly thinking about race,
race implicitly makes its way into these systems,
in ways that we simply have to understand.
One of the things I think we’re thinking about here at IDSS,
and certainly thinking about it at UT Austin,
it’s a really interesting question and I want to pose it to you,
because I think it’s one of the really interesting challenges,
a paradoxical question in some ways.
And that is: can you be “fair”,
and still not engage structural racism, for example.
What I mean by that
is even as we think about this notion of fairness,
treating likes alike, treating everyone equitably,
in terms of how these models function, how they create knowledge,
how they make predictions, how they calculate probabilities.
The idea is that in some ways, even if we're being fair,
we may not necessarily be dealing with the deeply embedded inequalities
that are structured by the world which we live in.
And so this notion of fairness in some ways, if we’re not careful,
makes the assumption that we all start from the same spot, right?
That we all have relatively access to the same resources
and the same opportunities,
and we know that that’s simply not the case.
And so even a fair algorithm,
in some ways may not be necessarily effective or adequate enough
to deal with the structural inequalities that we oftentimes hear about.
Now, social scientists have spent decades
literally thinking about the ways in which racism manifests itself
in many different ways.
They’ve identified different expressions,
different formations, different kinds of racism.
Here are just three, just to give you an example.
Now, interpersonal is something that I think most of us are familiar with,
the use of racial epithets, prejudicial attitudes.
Those are things that oftentimes are familiar to us,
things that we perhaps have seen in the news media,
perhaps things that we’ve seen in our own real world experiences,
and that’s something most of us get, right?
That tends to be tangible,
that tends to be something that we can see,
something that perhaps you’ve experienced.
But social scientists
have also thought about institutional and more structural forms of racism,
that tends to be less tangible,
less visible, therefore much more difficult to address,
because they oftentimes are happening in ways that we don’t even understand,
and happening in ways that are almost kind of on autopilot.
And I'll give you an example of that here in a moment.
So, think about a system like this, right?
And if you think about each of these different domains,
if it’s the credit markets, if it’s health, if it’s education,
these each represent different subsystems,
different domains, different institutional spheres
in the world in which we live,
and they each have their own different practices,
their own different policies, their own different customs, behaviors
and the ways in which they function,
that can put some at a structural advantage,
and others at a structural disadvantage.
And that’s what we mean in some respects by the notion of institutionalized racism.
Structural racism, and this has become important here at MIT
and IDSS, and the new College of Computing
and similarly at UT Austin.
Social racism is something that’s become more and more
of a focal point of conversation and concern
in society and politics and the tech sector,
and for many years, in academia.
And it’s this idea that racism and forms of inequality
are some ways that are deeply baked into the very functioning and fabric
of how society works and how it functions.
We can think about three conditions social scientists have identified,
in some ways, there are three conditions that make up a system of discrimination.
For example, in each of these subsystems, there’s disparity or inequality exists.
You can certainly look at each of these and say they absolutely do.
Another example might be, do some of these disparities or inequalities
exist precisely because of discrimination?
Again, you could argue
that in many of these instances, in fact all, that is the case.
But the third component or the third element,
which is really interesting, and really important
in terms of understanding,
the challenge and the complexity of what we mean by structural racism,
and that is when disparities in one of these subsystems
is reciprocally linked to disparities in other subsystems.
And this is, in some ways, both a theoretical interesting challenge,
but if you think about it from a computational perspective,
and this is why we’re dealing with it within the context of A.I.,
the context of these computational systems.
In other words, how do you design a model?
How do you design a computational system or algorithm,
that address is this interconnectedness,
of these different subsystems and domains,
and that’s a very complex challenge.
So this notion of fairness
tends to treat racism, or the problem of racism,
as primarily an individual problem, again, sort of interpersonal.
But when you start thinking about the institutional and the structural,
you begin to see where the computational challenges exist,
and how this will be something, or why this will be something
that’s going to require
this interdisciplinary combination of kinds of expertise
that we’re gathering here at MIT
and certainly that we're trying to do at UT Austin as well.
I’ll give you an example
of what structural racism is, we hear a lot about this term,
particularly as a result of the events of the last year and a half,
a lot of people have been trying to understand this,
creating reading groups, community students are coming to me,
perhaps if you’re a faculty member, your students are coming to you.
People are creating reading groups
all over the country and all over the world
to understand what we mean by this notion
of structural, systemic racism,
as it was becoming more popular and more common
as a result of the tragic events that have happened
over the last two years or so.
So take home ownership and credit markets, right?
So home ownership is actually an interesting
and a critical aspect to quality of life, right?
In some respects, for many Americans,
home ownership is their primary pathway to wealth accumulation.
But home ownership is also access to education,
access to better health, access to other kinds of resources,
social networks and things of this manner.
So if you think about credit markets and how they relate to each other,
you don’t own a home without good credit.
You don’t own a home without access to financial resources
that are made available to you via the credit market.
And researchers have found over the years
and they’ve been studying this for 40-50 years or so,
the kind of discriminatory practices
that are quite common in the credit markets, for example.
So if you think about this as a subsystem,
we think about this as a particular kind of institution
and the different forms and manifestations and expressions of discrimination
that function in this way.
The way in which some people get steered towards
subprime versus prime loans,
the way in which some people get steered to lower homes
and lower income communities,
as opposed to more middle class or higher income communities.
And so in this sense,
thinking about the ways in which we might understand
how this sub system as an institution functions
in ways that replicate inequality in a way that’s quite significant.
But then when you begin to think about this
in a sort of systemic or structural way,
you begin to understand that the complexity of this,
and why this is so profound, and why it’s so difficult to deal with,
both in a political way,
and certainly in a mathematical or computational way.
So here again, this relationship between credit markets and housing.
And again, when we look at the downstream effects,
of lower income households, lower income communities,
lack of access to high quality housing,
the downstream effects are quite significant.
And this is the fact one of the verticals that IDSS is looking at is home ownership,
and looking at structural racism in the credit markets,
and structural racism in terms of home ownership
and how this is in some ways a key anchor
in terms of how inequality,
and all of the things associated with inequality,
health challenges, lack of access to quality schooling,
lack of access to job markets, and economic opportunity and mobility.
All of these things become interconnected, and it’s this aspect, right,
the sort of structural and systemic nature of this,
that makes it such a vexing and complex problem.
Just earlier this week, I think, this article appeared in The New York Times
about a whole new generation of companies
that are trying to leverage Artificial Intelligence
to address some of these biases that have existed for so long
in the credit markets,
that have restricted people’s access to quality homes, quality housing
and the kinds of opportunities in life conditions and circumstances
that come along with that.
But if you read this article, the kinds of things
that these new generation financial loaning institutions are operating under,
how they’re using Artificial Intelligence,
what kinds of data are they using,
what kinds of problems are they solving for,
and how are they designing their models?
And when you read this article,
at least I did, left reading the article, somewhat concerned and somewhat disturbed.
The kind of data surveillance they are now operating,
this idea, that because so many people,
I think something like 15-20% of people
are just locked out of the conventional credit market system,
and so this become perhaps an alternative way
to getting access to these resources.
And yet they’re being exposed and rendered vulnerable,
to having to release all kinds of privacy,
of information about their financial situations,
about their banking account, about their lifestyles.
And so in this sense, subjecting them to what I would call
a kind of data surveillance that, in some ways,
could render them particularly exposed,
in a world in which privacy, data security and things of this nature
are becoming increasingly important.
Let me end with a couple of other things here.
Part of what we’re trying to do in the College of Computing,
and IDSS more specifically,
instead of thinking about the future of computational systems,
and how do we design them in ways
that attend to these structural inequalities.
I’ll admit it is a difficult challenge, but it is one, I think,
that my colleagues here and my colleagues at UT Austin
are committed to trying to grapple with, and committed to trying to address.
And oftentimes, we hear a lot about, you know,
well, who’s making these models, who’s building these systems?
We hear a lot about the pipeline, and who gets access to these resources,
who gets access to the knowledge and skills to build these systems.
And we know, that that’s a critical aspect of this, right?
We’re learning that the reason that a lot of these systems are faulty,
the reason that you have this facial recognition system disaster,
is because the training data sets are oftentimes not representative.
So in some ways, it leads to higher error rates,
where people with darker skin color,
men, women of darker skin color more specifically.
But there’s something else that I want you to think about,
it's not only who's building A.I. ,
but also who’s using A.I.
In some of the research we’ve been doing,
when we talk to different sectors in different industries,
take the public sector, for example,
those in law enforcement,
those in transportation, those in public health,
to what extent have they been adequately trained
to be sensitive to these issues around bias and discrimination?
To what extent were they adequately trained to understand
that perhaps the software they’re using,
the computational systems they’re using,
may not be based on a representative data set.
They may be designed in some ways that lead to these unintended outcomes
or unintended circumstances,
that lead to people being wrongly accused
for crimes that they didn't commit.
They leave people on the margins of our society,
even though that wasn’t necessarily your intent.
And nevertheless, it could happen
as a result of the ways these systems function and operate.
Let me leave you with this quote, I think it’s really representative,
in some ways it summarizes the inspiration for doing this work,
why I’m engaged in this work,
why I think my colleagues here, and at UT Austin are engaged in this work.
Mr. Williams here, when he was wrongly accused and arrested
because of a faulty recognition from facial recognition software,
when he was sitting with the police officers,
and they showed him the picture that they used to identify him,
he looked at it, and he said, “This obviously isn’t me. Why am I here?”
And the police officers looked at each other and said, “Well,
it looks like the computer got it wrong.”
And it turns out that not only did the computer get it wrong,
but those law enforcement officials got it wrong as well.
So this is the issue,
how do we design these systems,
in ways that not only mitigate these kinds of outcomes,
but how do we train the next generation of people
who will be using these systems,
to make sure that they make decisions that are ethical, responsible
and in some ways address these systemic inequalities?
Thank you.
(Applause)