Tzu-Sheng Kuo: Hi, my name is Tzu-Sheng. I am presenting our CHI 23 paper on
understanding frontline workers’ and unhoused individuals’ perspectives on AI
used in homeless services. This is collaborative work between Carnegie Mellon
University and the University of Pittsburgh. AI systems increasingly
influence high-stakes decision-making in public services, including criminal
justice, child welfare, and public school assignment. However, a growing body of
research in HCI demonstrates that these AI systems are often not designed based on an
understanding of key stakeholders' desires and concerns. While much of this prior HCI
literature has studied contexts mentioned in the previous slide, the rapid spread of
AI systems in homeless services has received surprisingly limited attention
from the HCI community. In this project, we focus on homeless services, a context
where the use of AI-based decision-support systems is rapidly spreading to cities
across the world. Yet, we lack an understanding of impacted community
members' perspectives on these systems. For example, such systems have been
considered or deployed in US counties, including Los Angeles, San Francisco
Allegheny County in Pennsylvania, and Ontario, Canada. In this project, we study
one of these AI systems deployed for over two years in the United States. Throughout
our paper and the presentation, we refer to this AI system as HAA, which is a
pseudonym for the housing allocation algorithm. HAA is an AI-based
decision-support tool that the county uses to prioritize housing programs for people
experiencing homelessness. HAA works as follows: The housing assessment process
starts with an applicant getting in contact with either a county’s link staff
in the office or a frontline worker in the field unit. Once an applicant gets into
contact, the county workers pull the applicant's personal information from the
county's data warehouse. Based on this information, HAA predicts how likely the
applicant will experience these situations if they remain unhoused over the next 12
months. Based on this likelihood, HAA generates a risk score between 1 to 10.
The higher the score, indicates the higher risk of being harmed due to homelessness.
The applicant then gets assigned to the housing waitlists corresponding to their
scores. Once these housing programs have openings, the county connects people with
the downstream housing providers. In rare cases, county workers may run the
alternative assessment or file a prioritization request when they believe
the score doesn't reflect an applicant's vulnerability. In this diagram, the
highlighted area is where the HAA system mainly operates. Our research goal is to
understand community stakeholders' perspectives on the design and use of
AI-based decision-support tools in homeless services. In particular, we
target stakeholders who interact most closely with or are most affected by the
system, but whose voices have yet to be heard. These stakeholders include field
unit workers who have direct experience using HAA on the streets, service
providers who have a birds-eye view of HAA’s downstream impacts, and people with
current or former experience of homelessness, who are directly impacted by
HAA’s decision. In the remainder of the talk, I will briefly go over these four
sections, starting with our feedback elicitation method. We aim to elicit
detailed feedback across the entire AI development pipeline from stakeholders
spanning a broad range of backgrounds. To do so, we introduced AI lifecycle
comicboarding, an adapted comicboarding method uniquely tailored to understand the
sociotechnical implications of AI systems. Specifically, we first identify eight
common stages in AI's design and deployment pipeline based on prior
literature. For each stage, let's say model definition, we create a set of
“story starter” panels to set the context for a key design choice. The next panel
then presents a blank space, along with an open-ended question to prompt ideation and
critique. The final panel reveals how the given aspect of the system is currently
designed in reality. Using this approach, in the current study, we developed a set
of comicboards to understand frontline workers’ and unhoused individuals’
perspectives on AI used in homeless services. All our comicboards are shown in
our paper. Importantly, we based the information in our comic boards upon
detailed technical reports published by the county about the HAA system. Moving on
to study design, we recruited four workers in the county's field unit, five
non-profit service providers, and 12 people with former or current experiences
of homelessness. Our study started with a 15-minute semi-structured interview to
understand the participants' backgrounds, followed by an hour of the AI lifecycle
comicboarding activity, and concluded with a 15-minute reflection. The study took an
average of 108 minutes. We provided participants with a $60 compensation. We
analyzed the transcript through open coding and affinity diagramming. In total,
this process yielded three third-level themes discussed next. Now, let's look
into our findings. We organize our findings around three third-level themes
identified through our analysis, including desires for feedback opportunities,
feedback on HAA’s design, and feedback on HAA’s deployment. Overall, we found that
while both unhoused individuals and frontline workers wished for opportunities
to provide feedback on the AI systems, there was significant skepticism within
the county that non-AI experts could provide meaningful feedback. However, our
findings demonstrate that community members can provide specific and critical
feedback on an AI system’s design and deployment based on their observations of
systematic limitations and errors in HAA’s predictions. Given the limited time
available for this presentation, I will focus on sharing participants' feedback on
HAA's design. Some participants challenged HAA’s problem formulation: the fundamental
idea of prioritization. They suggested the county focus its resources on addressing
the root problems of homelessness rather than prioritization. Participants also
perceived that HAA had been designed to reflect the county’s values rather than to
reflect local community members’ needs. For example, a service provider perceived
that HAA is measuring the financial cost to systems that have power rather than the
actual harm to unhoused people. Participants also shared several concerns
about the validity of the HAA model. For example, some participants were concerned
that the proxy outcomes that the HAA model is trained to predict, such as emergency
room visits, could be highly misleading because people go to the emergency room
for various reasons. They were also concerned that the score could not
accurately reflect who was in more urgent need of help. For example, one service
provider mentioned that someone with years of chronic homelessness might be safer on
the streets than a young white female. However, she might receive a zero or one
from HAA because of the lack of records. Participants worried that HAA’s
predictions might become less and less valid as time progresses because they
believed the relationship between the model’s inputs and the real-world outcomes
that it predicts is highly unstable across time. For example, a frontline worker
mentioned that they are now proactively reaching out to people who are more
vulnerable. Therefore, the vulnerability changes. As mentioned earlier, HAA relies
on administrative data in the county’s data warehouse. Some participants worried
HAA misses the data of people who are averse to accessing public services due to
institutional violence and prior traumas. They argued that the lack of engagement in
services is not a lack of need. Participants were also concerned that the
administrative records could be outdated, as a person's situation could quickly
change. Finally, participants emphasized that it is critical to consider
qualitative narratives when making decisions about housing prioritization,
given that many social dynamics on the streets cannot be digitized. In summary,
participants questioned whether the HAA system helps address the root problem.
They also perceived that the system optimize toward the county's interest
rather than community needs. For the model, they raised concerns about its
misleading proxy outcomes, unreflective scores, and risks of dataset shift.
Finally, they argued that the data could be imperfect and, therefore, qualitative
narratives are critical. As mentioned earlier, we only reviewed selected
feedback on HAA's design in this presentation. Please check our paper for
more details on other findings. Now, I will conclude the presentation with a
reflection on our study. First, as demonstrated in our findings, our method
is able to surface hidden assumptions and open up conversations around AI's design
and deployment. Second, it can also elicit detailed and critical feedback across the
AI development lifecycle. Overall, our findings demonstrate that community
stakeholders can provide specific and critical feedback on AI system designs, if
they are empowered to do so. With that, I would like to end the presentation by
thanking our participants for their valuable time and feedback. Thank you.