On Data Cascades in High Stakes AI, done in collaboration with my colleagues Shivani Kapania,
Hannah Highfill, Diana Akrong, Praveen Paritosh and Lora Aroyo all at Google Research.
Data is the critical infrastructure necessary to build AI systems.
It can determine performance, fairness, robustness, safety, and scalability of AI systems.
Paradoxically, for AI researchers and developers, data work is often the least incentivized aspect,
viewed as ‘operational’ relative to the lionized work of building models.
Data is often viewed as a databases, legal, or licensing issue. We believe that HCI
has an important role to play in elevating data as a first class citizen of AI systems.
In this paper we examine data quality practices and challenges among AI practitioners working on
high stakes AI, where there are safety impacts on living beings, such as maternal health,
road safety, and climate change. Our primary contribution is to present
empirical evidence and conceptualization around data cascades in AI systems.
Our method involved semi structured interviews with 53 AI practitioners working in high stakes
domains, including engineers, researchers and data scientists, based out of India, USA,
and Eastern and Western African countries.
We observe and define Data Cascades as compounding events causing negative,
downstream effects from data issues, that result in technical debt over time. In our
study, 92% reported experiencing at least one cascade. 45% experienced two or more cascades.
Data cascades are complex, long-term, occur frequently and persistently in our study.
They are opaque in diagnosis and manifestation.
They are triggered when conventional AI practices are applied in high-stakes domains,
like viewing data as operations, moving fast to proof of concept, hacking model performance without
consideration for data quality, or undervaluing domain expertise and data labour.
Data cascades have negative impacts on the AI development and deployment processes, leading to
harm to beneficiary communities, burnout of relationships, discarding entire datasets, and
performing costly iterations. Cascades are often avoidable by step-wise and early interventions
in the development process.
We observed four types of data cascades. I’ll discuss two for brevity.
High stakes AI systems are typically brittle, closely interacting with previously undigitized
environments, like air quality sensing, ocean sensing, or ultrasound scanning, leading to even
more reasons for a model to break—like limited training data, complex underlying phenomena, or
volatile domains. Data cascades here appeared in the form of hardware,
environmental, and human knowledge drifts. As an example, even the slightest movement of a camera
due to environmental conditions can result in failures in detecting traffic patterns.
Artefacts like fingerprints, shadows, dust on the lens, improper lighting,
and pen markings were reported to affect predictions. Conventional AI practice of training
on pristine data, but deployed in the messy real world appeared to trigger these cascades.
Another type of cascade was triggered when AI practitioners were responsible for data
sense-making in contexts in which they did not have domain expertise, like diagnosing
fractured bones, identifying locations that could be poaching targets, and congenital
conditions leading to preterm babies all depended on expertise in biological sciences, social
sciences, and community context. Conventional AI practices like overt reliance on technical
expertise and unsubstantiated assumptions of data reliability appeared to set these cascades off.
Data cascades point to the contours of a larger problem: residual conventions and
perceptions in AI/ML drawn from worlds of ‘big data’—of abundant digital resources; of
model valourisation; of moving fast to proof-of-concept; and of viewing data as grunt
work in ML workflows We need to move towards a proactive
focus on the practices, politics, and values of humans of the data pipeline. We need to move
from the current approach of goodness of fit to goodness of data. From doing more to doing better
with data. We need to innovate on structural incentives to recognize and reward data work, in
conference tracks, organizational recognition of data gluework, through greater
collaboration and shared benefits for ML developers and field partners. Data
ethics and practical data work, oversight boards like IRB, and ethics standards
should be a part of AI education and praxis.
Finally we need greater data equity in the Global South.