Hi everyone.
My name is Hao-Fei and I'm
excited to talk about our work on
soliciting stakeholders fairness notion
in child maltreatment protective system.
I am presenting this work on
behalf of my wonderful collaborators,
Logan, Ruiqi, Paige, Alex, Steven, and Haiyi.
With the widespread use of machine learning algorithms in
various high-stake decision-making scenarios.
The fairness of these algorithmic system
has drawn increasing attention.
Fortunately, the machine learning community has
proposed multiple approaches and criteria for fairness,
which can be broadly classified
into group fairness, which requires
statistical property be equalized across groups.
And individual fairness, which requires
similar individuals be treated
similarly by an algorithm.
However, while these fairness criteria
provide guarantees on their own,
they are proposed by machine learning
practitioners and the stakeholders
were not involved in the development,
development of these criteria.
Therefore, the fairness definitions
may not reflect the viewpoints of these stakeholders.
At the same time. These stakeholders are also
the ones that will be affected by
the outcomes of the algorithms.
The main research questions of
our study is therefore finding out
which fairness definitions align
with the stakeholders viewpoint.
To answer this, we developed a framework to elicit
the subject fairness notions
of non-expert stakeholders.
Our framework consists of two parts—
an interactive interface that
allows stakeholders to examine
the data and audit the prediction made by the algorithms,
followed by an interview protocol that probes
the stakeholders' believe on fairness of the algorithms.
Since fairness definitions are often complicated,
the combinations of an interactive interface and interview
will help stakeholders visualize
and understand these definitions.
We use child maltreatment predictive
systems as the context of our study.
Each year, there are millions of
child maltreatment referral cases in the US,
and call screeners have to make
the decisions between screen-in for
investigations or screen-out if the cases are low-risk.
Child maltreatment prediction is a high-stakes scenario,
and a machine learning-based risk assessments
tool was recently deploy in Allegheny County,
Pennsylvania to assist with the decision-making.
The algorithm is trained on past child referral cases
using features such as
the age of the child and the information.
of the perpetrator. The algorithm
predicts the risk level of each case, and
call screeners make the final decision
after seeing the output from the algorithm.
We conduct a study with
stakeholders of the child maltreatment predictive system,
which consists of four parents and eight social workers.
The interviews follow our fairness elicitation framework,
which I will now go over.
Recall that our framework had two parts,
an interactive interface and an interview protocol.
The first part of our interface is the group view,
which allows stakeholders to define
groups and subgroups that they are concerned about,
and compare the algorithm performance
across different statistical metrics.
After seeing the algorithm performance,
we asked the stakeholders
if the group fairness approach is
appropriate for the child maltreatment situation.
For example, we ask if a fair algorithm should
make high-risk prediction at
the same rate or victims of different races,
which correspond to the notion of statistical parity.
Another interface is the case-by-case view,
we show stakeholders pairs of cases at a time and
stakeholders can inspect and
compare detailed features of each case.
This correspond to the individual fairness notion.
We first show stakeholders cases without telling them
the decisions made by the algorithms to collect
their fairness notions regardless
of the predictions of these outcomes,
and understand the factors that they will
consider when evaluating the cases.
We then interview them by asking how they will
prioritize the two cases for each pair.
Whether they should receive the same decisions
or if one of the two cases should be prioritized.
Here are some highlights of our results.
We found out that equalized odds was
the most preferred group fairness
approach among the stakeholders.
One of the reasons is that equalized odds
align with the assisting fairness belief of
these stakeholders, which is for
the accuracy of the algorithm be
as high as possible, and as
even as possible for all groups of people.
At the same time, even though fairness
is what stakeholders desire,
they are also not willing to trade
accuracy in order to achieve that.
For individual fairness, we found out that
there are no unanimous agreement amongst stakeholders.
For most situations, we see
contentious opinions on how to prioritize the cases.
We are able to identify
a few reasons for the lack of agreement
for individual fairness.
First of all, stakeholders disagreed on
the implications of several
key features in child maltreatment,
such as the victim's age. At the same time,
stakeholders also use
fundamentally different heuristic to compare
individual cases which lead to
different opinions on how the cases should be prioritized.
Those are just some highlights of our work.
You can find all the details in our paper.
If you have any questions,
feel free to email me and I would love to chat.
Once again, I would like to thank
all my collaborators for making this possible.
Thank you all.