Hello, my name is Katherine Zellner, and I'm a member of the
interactive systems for healthcare lab formed by Drexel
University, Children's National Hospital and records University.
Today I'll be presenting our research on developing a speech
based model for tracking the progress of activities during
extreme action teamwork.
Our labs research is situated within emergency medical
teamwork, specifically the area of trauma resuscitation. During
trauma resuscitation, a complex team of clinicians, including
surgeons, anesthesiologists, nurses, and more work to
stabilize severely injured patients before moving them to
further care. During trauma resuscitation, medical delays
and errors contribute to nearly half of preventable deaths,
meaning there's room for a system to help improve the
teamwork in trauma resuscitation.
Our team's goal is to develop a system to support awareness of
time delays, and the progress of the team during emergency
medical scenarios, and hopes that will reduce errors and
delays. To create the system we need to be able to automatically
and unobtrusively monitor activity progression.
We hypothesize that speech is a data input that can be used to
identify the stage of activity performance, ie during before or
after.
In this paper, we developed the model to determine the temporal
progression of activities based on speech and within trauma
resuscitation.
related work in statistical language modeling explores the
use of narrative schemas to represent script knowledge and
predict a missing or upcoming event in simple stories about
daily activities. We use the concept of narrative schemas,
and we model stories through through them, but our work
differs in three ways. First, we're using stories derived from
actual conversations during dynamic teamwork. We use speech
data to infers this team's status and progression of work,
rather than describing or generating stories about the
work. Lastly, the stories in our dataset are often incomplete,
and composed of choppy, grammatically incorrect
sentences, which makes the machine learning model training
more challenging.
To accomplish this study, we situated the study within the
medical emergency medical domain, and we transcribed 104
audio recordings of actual trauma recitations. Using the
transcripts, we first study the nature of speech during 34
clinically relevant activities.
25 of these were assessment activities where the clinical
team is evaluating the patient. And nine of these are control
activities when the team is actively trying to control the
status of the patient.
From this analysis, we identified 11 communicative
events across the three stages of activity performance before
during, and after.
Here we can see the communicative events that we
derived, and the will for stage also known as the activity
planning and preparation stage. clinical teams both discuss
assessing feasibility or need for activity, they'll request
members to perform an activity. They'll request clarification or
more information on an activity. They'll inform the patient about
an activity or request them to perform an action.
The during stage or the activity execution stage, clinicians are
asking patients questions or requesting patients to for them
to perform an action. Clinicians are requesting information on
the progress of the activity from each other. They are
reporting progress if an activity they are requesting to
modify or terminate an ongoing activity. Then the after stage
is the evaluation stage, where clinicians are reporting results
or outcomes and activity. They're requesting clarification
or more information about the finished activity. And they're
assessing results or the outcome of an activity.
These communicative events served as building blocks for
our narrative schemas. And we developed a narrative schema for
each of the 34 clinical events. This is an example of one such
schema that was used for assessing the airway. So first,
we can see that there is a request to perform airway
assessment or clinicians discuss check the airway, is the area
projected Is there a good then they will move to carry patients
are requesting them to open their mouths with speech lines
such as, Hey, do you know where you are? Wake up wake up how you
feeling? Can you give me your name Open your mouth. Then the
after stage will be when the report results of the airway
assessment and clinician will say airway is an intact airway.
This page and arrows are protected. So by analyzing these
speech lines, we can tell what stage of activity progression
they are in.
We developed the final generalized speech based model
by extracting and aggregating the generalized aspects of those
34 schemas. The speech flow model here illustrates the
differences between both assessment and control
activities. It starts on the left showing the progression of
assessment activities. And then on the right, you can see the
control activities. Assessment here generally is requesting an
activity assessment, then interacting or requesting
perform actions reporting progress, reporting results and
activity, but in control, first, they assess the feasibility or
the need of a control activity. Then they recoup requested
report progress report outcome, and they will repeat these steps
until the activity is over.
We then evaluated this model by applying it to 70 new
transcripts to see if it held true a researcher read through
each each line and saw if they could
apply the model to evaluate when in this each model they were
happening so for Okay, can you check his pupils was before
stage look at me don't move because during pupils are two
millimeters reactive through reporting an outcome and a
result. So that's after.
In this evaluation, we found that the model of reliably
recognized an activity stage and 98% of activity related
conversation instances showing that this model is suitable to
be applied to our trauma resuscitation example.
Our results have several implications for designing
technology to support complex teamwork, and specifically to
detect delays during time critical teamwork. First, the
system design should focus on the after stage, we found that
it is the most critical role to detect activity completion.
Secondly, the majority of speech lines took place in the after
stage. And this means that there are fewer in the pool for and
during stages. And we may require other modalities to
detect those stages such as motion or equipment sensors.
Lastly, we found that incomplete speech still poses a challenge,
as it would be difficult to automatically detect activity
progression if we do not have complete sentences. This is a
dense and rich paper with a lot more information about every
step. But specifically if you would like to learn more about
how we apply this model to machine learning applications,
there is so much more there that you can read more about in the
paper.
Thank you so much for listening. I'm really excited to share this
research with you all. Feel free to send us any questions to
email. And thank you so much