Hi, I'm Emma Beede and today I'll be presenting our paper: "A Human-Centered
Evaluation of a Deep Learning System Deployed in Clinics for the Detection of
Diabetic Retinopathy" on behalf of my co-authors: Elizabeth Baylor, Fred Hersch,
Anna Iurchenko, Lauren Wilcox, Dr. Paisan Ruamviboonsuk, and Laura Vardoulakis.
Diabetes is a growing problem around the world, and with diabetes comes
complications, including diabetic retinopathy, which we refer to as DR. DR
is a condition caused by chronically high blood sugar that damages blood
vessels in the retina, the thin layer at the back of the eye responsible for
sensing light and sending signals to the brain. These blood vessels can leak or
hemorrhage, causing vision distortion or loss. DR is one of the leading causes of
vision impairment in the world and causes 5% of cases of blindness
worldwide, excluding refractive errors. In Thailand in particular, 34 percent of
diabetic patients have low vision or blindness in one eye. And there is a
problem of access to care: there's a shortage of specialists that are hard to
get to, and it takes a long time for a patient to get the results back from
their eye screening. For this study that we're going to talk about today, our team
developed a machine learning-based model to detect diabetic retinopathy and
diabetic macular edema on standard fundus photographs and make an
assessment of whether or not the patient needs to be referred to a specialist for
treatment. Two retrospective evaluations have shown that the system performs very
well, essentially on par with a retinal specialist. In a partnership with the
Thailand Ministry of Health, the deep learning system was deployed in eight
clinics across Thailand in a year and a half long prospective study with about
7600 patients. For our paper, we describe a human-centered evaluation of that deep
learning model that we conducted prior to and in parallel with this large scale
prospective study. Before we talk about the design of our study and the results,
I want to review some background on AI within healthcare.
AI technologies, including deep learning are increasingly being applied to tackle many challenges
in healthcare including breast and lung cancer screenings, tuberculosis, and
diabetic retinopathy. However, one thing we do know is that these systems often have
issues moving out of an R&D environment and into a clinical environment:
clinicians might not find them useful, there are challenges when it comes to
integrating into current systems and workflows, and they often make the
workload for clinicians worse, not better. And in the worst cases, these systems in
the wild don't improve clinical accuracy.
One possible reason for this is that there is currently no specific mandate for AI systems to be evaluated
through human-centered observational clinical studies nor is it common practice.
Prospective studies are beginning to become more common as a way
to evaluate deep learning models within a clinical environment. These studies are
an excellent step forward and are designed to provide additional evidence
of model accuracy. But it's important to note that they are not sufficient to
truly evaluate impact on patient care, nor are they designed to uncover
socio-environmental factors that impact model performance in the wild.
There is though, a long tradition of HCI studies that examine environmental and
contextual factors surrounding systems designed for a clinical environment,
including work from Hartswood, Alberdi, and others. And human-centered
evaluations of deep learning systems in health care is a growing area of
research. Recent research from Cai, Yang, and others has shown progress in the area.
But the CHI community has faced many challenges when trying to conduct
research on deep learning in medicine. As Yang and colleagues discussed in 2019,
researchers often face difficulty fully embedding into clinical workflows. As a
result, studies and evaluations are often lab-based and lack the ability to fully
explore the technology in context. Similarly, there are many challenges that
come with conducting studies and involve patient data in a clinical setting. Even
with informed consent and appropriate protocols around this data,
HCI researchers face challenges evaluating systems on authentic data and
often are limited to conducting evaluations solely on synthetic or mock
data – which may or may not be sufficient to fully evaluate the technology at hand.
So to move on to our research – we know that for our model to be successful
it not only needs to be accurate, it needs to meet people's needs. In this
case, we had to make sure this tool was going to work for the nurses that would
be using it every day and the patients that it would service. To do this, we took
a human-centered approach. Alongside the prospective study evaluating system
accuracy, we conducted a human-centered study which involved field research at
11 clinics across Thailand over a period of eight months. This research took place
both before and after the deployment of the deep learning system. Our data included
interviews, observations, as well as logs analysis and usage data of the system.
Today I'll review some of the findings from our research. For the rest,
I encourage you to take a look at the full paper.
Before we talk about the results of the deep learning system, I want to give an overview of the general
eye screening process that takes place at the clinics we visited. We observed some
differences across clinics but in general there was a consistent workflow.
100+ patients would queue for screening in the morning to receive a
full diabetic checkup – checking their blood sugar, as well as their feet their teeth and
their eyes. They would sit with a number and wait to be seen because
there were no appointments. The whole experience could take up to five hours.
For each patient, the eye screening portion is only allotted 90 seconds to
take the picture and to do a quick assessment to decide if the image
needs to be sent to an ophthalmologist for review. It often takes longer, which
causes delays and the long waits. This process repeats each screening day. The
images are then sent in batches to an ophthalmologist for review on a CD. These
were sent anywhere from every week to every two months depending on the number
of patients seen at the clinic. Once the images reach the ophthalmologist,
patients would receive the results in about a week. It was our goal to cut down
on this time and give them the results in real time, which would help them get to care sooner.
One thing we wanted to understand before nurses started using
the system was their expectations for AI assistance with eye screening.
Participants told us that they saw the potential of the AI to help them up-level
their skills. It could help them distinguish between the different levels
of severity of DR. In some cases, it would confirm what
they already knew -- it would allow them to assess a patient without a doctor's help,
giving them more autonomy. At the same time, they were concerned about the
amount of time it would take to upload photos due to internet issues at the
sites. Any amount of time added when screening hundreds of patients would be
costly and could cause patient delays and cause them to be rescheduled.
After we deployed the system, we went on site to see how well it was working.
One of the first things we learned had to do with something called gradability.
Gradability refers to the ability to read an image and make an assessment.
When clinicians are conducting screenings, they have guidelines that
they use to determine if the photograph of the retina is clear enough to make an
assessment about whether or not the patient has DR. These guidelines and
their interpretation can vary between clinicians. Just like clinicians, the deep
learning model needs to set a threshold for features around image
quality like blur and darkness before it will make an assessment about DR based
on that image. If the image is extremely blurry, just like the clinician the model
may call it ungradable. After the deployment of the deep learning system,
we observed from system logs that on average, twenty percent of images
couldn't be read by the model. During our fieldwork, we observed several causes for these ungradable images.
Because of the high volume of patients seen at the
clinics, there was limited time to align the patient to the camera to take a good
image. And to take a good image, a patient's pupils need to be large. This
was being affected because screenings were taking place in brightly lit rooms
because nurses were sharing space where teeth exams or patient consultations
were also taking place. When the nurse takes the first image, there's a
bright flash to take the photo. This causes the pupils to constrict
which makes it really hard to get a gradable image of the second eye. In
addition, they were using older cameras, some of which needed to be repaired.
Because the image quality threshold for gradability didn't match nurses
previous practices of grading, it caused some frustration. They were used to
doctors in Thailand that were willing to read all images, even if they
were too dark, because they're able to look at the whole case of the patient
and make an informed decision on how likely it is that that patient has DR,
whereas our tool only receives and makes assessments of DR based on fundus
photographs. The ungradability rate caused nurses to think that it wasn't as accurate.
We also found challenges with our protocol. Our protocol stated that in
the case of an ungradable image, to be on the safe side, a patient should be
referred to an ophthalmologist. But that ophthalmologist was at a
hospital an hour away and many folks couldn't afford the transport cost
or the time to take off work to go. Designing the protocol this way had
downstream effects. 20% of people were being referred for an ungradable image
and were made to travel. Previously, they simply had to wait a few weeks to have
their image read. We also observed nurses dissuading participants from joining the
prospective study if they felt that the patient might be unnecessarily burdened by this referral.
As a result of this research, we were able to repair the
cameras that needed to be fixed, and we added a curtain or a drape to the camera
setup. In addition, because of this learning we made changes to the research
protocol to wait until the grade was received back from the ophthalmologist
before referring. In order to address the trade-offs of using the AI tool, we found
that using a combination of AI and a human clinician was best. This model is
also in active development and there are continued improvements to gradability
and other performance metrics.
So moving on to discussion. There is a rich body of work in HCI that has looked at human interaction
with computing systems in clinical settings. But deep learning systems surface new challenges.
On top of that, HCI researchers often face hurdles conducting studies of emerging
technology in the clinic so lab studies of AI systems are more common. But in a
lab setting, researchers can miss out on those socio-
environmental factors that surround use of the system. But there's good news: we
do see observational prospective studies of AI systems becoming more common.
These studies validate real-world clinical accuracy of deep learning
models in environments of use. We see opportunities to pair HCI studies with
these prospective studies. This has many advantages. Researchers can evaluate
using live data with target clinicians in a contextual environment and identify
socio-environmental factors ahead of widespread deployment, where issues may
arise from system use or misuse, changes to workflow, and effects on patients.
We recently completed our study and our target enrollment of 7600 patients for
the prospective study. We're currently analyzing the results of the prospective
study and we'll be able to share more in a future publication. Since this research,
we have held participatory design workshops with people from future
deployment sites including nurses, potential camera operators, and retinal
specialists, the doctors that would be receiving referrals for patients from
the system. In these workshops, clinicians designed new workflows that involve the
system and are proactively identifying potential barriers to implementation.
To wrap-up, the things we'd like you to take away from this research are that: first
accuracy is not enough. We often think of complex decision-making algorithms as
having the potential for unforeseen consequences, but we show that even in the
case with an simple image assessment tool
there can be downstream effects for people. Secondly, much of the success may
continue to be influenced by protocols around the use of the technology, as we
saw in our prospective study. For the second half of the study, the
combination of AI plus eye specialists to review ungradable images
was working well. It still allowed the majority of patients to get results in
the moment and relied on "old" processes when needed. And finally, we see the study
as an important example of how human-centered design fits into the AI
development process and how conducting human-centered evaluations before, during,
and after deployment of a system can not only inform development of the system
but also provide a critical understanding of the potential obstacles
and opportunities for workflow integration, adoption, and user trust. All
of which are essential to building tools that positively impact patient care.
This work was possible because of a large team of people at
Google, Rajavithi Hospital, and the many
clinics we visited across Thailand. We are incredibly grateful for their support towards this research.
For more information on this work, please see our
full paper. Thank you.