Leah Kulp: Hello and thank you for taking the time to view this presentation I'm Leah called a PhD student at Drexel University in Philadelphia, Pennsylvania, and this work was done in collaboration with
Leah Kulp: doctors and medical experts at Children's National Medical Center and Washington DC. The goal of this work is to understand factors contributing to checklist non compliance and how we can better design systems to support compliant use within the context of trauma recitation.
Leah Kulp: Just a quick presentation overview. I'll start with the motivation for this work provided domain background talk through the methods we used
Leah Kulp: Discuss the findings and the discussion from these findings and then how we can reconsider checklist design and some design recommendations and finally a conclusion.
Leah Kulp: So the motivation for this work in terms of checklist compliance. This can be defined in many ways compliance could mean adhering to protocols and making sure that all
Leah Kulp: tasks are performed by the team compliance could also mean compliance with the checklist and making sure that all of the boxes are checked off.
Leah Kulp: And studies have measured this in different ways. Some studies have found that checklist do increase compliance with protocols, meaning more tasks are performed with the presence of a checklist. However, other studies have found that checklist compliance does not always
Leah Kulp: Correlate with task performance. So one study found that 84% of items were checked, but only 54% of the tasks were actually completed.
Leah Kulp: And studies have documentation and timeliness, such as on electronic health records EHR.
Leah Kulp: Introduction of these systems has helped support collaboration and communication, but again we see misalignment between the design of the systems and actual practice.
Leah Kulp: Leading to work arounds and delayed performance. So once that he found only 8% of tasks were performed in real time. And most of these tasks were actually delayed.
Leah Kulp: Non compliant checklist using these settings may provide false reassurance that tasks have been performed or that tasks haven't been performed when they actually have closing widespread risk to safety. So what can we do about it.
Leah Kulp: We need to first understand the factors that contribute to noncompliant use of the systems and checklists and then design features to better support dynamic work prophecies
Leah Kulp: First, I want to give a background of the domain. So trauma resuscitation is a high risk time critical process of treating patients with acute injuries that require medical attention.
Leah Kulp: So the goal is to stabilize the patient identify any life threatening injuries and develop a plan for patient hospitalization.
Leah Kulp: This process requires teamwork and is guided by a state or process called a less advanced trauma life support.
Leah Kulp: This includes two sections. The primary survey of the A B C D E to quickly assess major physiological systems and then the secondary survey, which is a head to toe physical exam of any other injuries.
Leah Kulp: When a trauma patient comes into hospital into the trauma room in the emergency department, a team of seven to 15 members assembles to
Leah Kulp: Take care of the patient and has to have different roles and responsibilities. One of the roles is the team leader. This is a hands off or someone who stands at the foot of the bed and guides the team to perform all tasks and helps come up with the care plan at the end.
Leah Kulp: To support compliance with this at last protocol we created a paper checklist in 2012 with all of the steps on the ALS protocol that has been in practice.
Leah Kulp: And used since 2012 and 2017 we introduced a digital checklist. As you can see on the right team leader using the digital checklist in the trauma room.
Leah Kulp: And team leaders have been using this for the past three years.
Leah Kulp: So the digital checklist design just quickly was translation of the paper checklist. There's several sections on the paper checklists that are now tabs on the digital checklist.
Leah Kulp: We saw that leaders were taking lots of notes on paper checklist so we afforded this feature on the digital checklist with a margin area for pre hospital notes and then individual notes and areas to enter patient values.
Leah Kulp: When the checklist is submitted creates a law with timestamps and all the interactions that we can then use for data analysis.
Leah Kulp: So our overall research question in this study is what factors contribute to compliance and timeliness of checklist interactions.
Leah Kulp: To do this we first started by reviewing videos and generating activity logs. So what does this mean
Leah Kulp: We watched videos of actual cases and marked the start and end times of every single activity that was happening.
Leah Kulp: There's also the instance of this activity, whether it's the first time it's been occurring or the second time or more, and the provider that is performing the activity. We did this for 98 cases.
Leah Kulp: We also matched checklist items with tasks.
Leah Kulp: In these activity logs. So we matched 32 checklist items to the corresponding tasks. Some of these were one to one start and end times for the activities and checklist items. However, some of these were a little bit more complicated requiring multiple
Leah Kulp: Tasks for the checklist item to be considered complete
Leah Kulp: We also gather contextual information that is stored in a hospital database that includes information about the patient such as their age, the team activation level, the mechanism of injury and other contexts attributes.
Leah Kulp: To analyze the data we first wanted to determine checklist compliance. So we looked at the activity logs to determine whether tasks were started
Leah Kulp: Started and performed all the way to completion started but not actually completed based on our data dictionary or just not performed at all. And then we compared these to the checklist to see if the item was checked or not checked. When these different tasks performance were occurring.
Leah Kulp: We also compared timestamps between the activity logs and the checklist logs to determine the timeliness of these checklist interactions.
Leah Kulp: To take it a step further with the timing of the check offs. We also compare this to the contextual information that we had to determine if patient severity impacted the timing of the checklist check offs.
Leah Kulp: And finally we conducted qualitative video review to provide a deeper understanding of what's actually happening when non compliant behaviors are occurring.
Leah Kulp: So I'm going to present the findings and three sections that are related to the three sections of
Leah Kulp: Our methodology. So first we found five checklist compliant behaviors we have just depending on whether the task was performed to completion not performed or started not completed and then looking at whether the checklist item was checked or not checked.
Leah Kulp: So first I want to talk about the compliant behaviors. So this is what we would like to achieve 100% compliance.
Leah Kulp: Would be a true check which is when the task is performed and the item is checked off. So we found that 79% of all checked off items were true checks.
Leah Kulp: Most commonly, these were secondary survey exam items that had to do a physical exam and primary survey items.
Leah Kulp: We found that the true non checks. So this is on the task is not performed and the item is not checked again a compliant behavior 4% of these were true non checks and this actually occurred on the pre arrival items and the primary survey items.
Leah Kulp: For the non compliant behaviors we found three different types of behaviors. First, we're going to talk about false checks. This is when the task is not performed. But then the item is checked off.
Leah Kulp: We found that this happened for 16% of all checked items. And the most common ones were ocular integrity, which is an evaluation of the eyes on the secondary survey.
Leah Kulp: confirming that the patient has an IV placed and the nose exam on the secondary survey.
Leah Kulp: From the video review, we have some qualitative observations. So first, we found that those secondary survey items are often grouped with other exams.
Leah Kulp: For example, when the provider is checking the face. They might not necessarily look at eyes, nose, mouth individually. They might just say the facial bones are clear no injuries on the face.
Leah Kulp: Secondly, some of the tests are being done pre hospital. So for the IV access. Sometimes the patient comes in with an IV. And so this box is checked off by the leader, but this is actually not coded because it's not performed in the trauma room.
Leah Kulp: Leading to a false check
Leah Kulp: The second non compliant behavior that we found were inaccurate checks. So this means that the task was started, but not performed all the way to completion.
Leah Kulp: And then the corresponding item was checked off 5% of all checked items were inaccurate checks. The most common ones were checking the pulses, which requires checking pulses and four different sides of the body.
Leah Kulp: And then again on the secondary survey the mouth exam and the nose exam from our video review, we found that the pulses might not be completed.
Leah Kulp: Because this requires checking all four pulse areas and if another provider is trying to get IV access or check the blood pressure on one of the arms the bedside provider might just skip checking the pulse on the upper extremity.
Leah Kulp: Secondly, we found that the secondary survey exams tasks might be performed, but not verbalized so our ground truth activity logs require that these activities.
Leah Kulp: The findings from these activities be verbalized like the mouth has clear. But as I mentioned before, sometimes the secondary survey exam items are performed as a group.
Leah Kulp: And the provider will just say facial bones are clear the faces no injuries. Instead of looking at the mouth and nose individually.
Leah Kulp: Finally, for non compliant behaviors we found that there were failed checks. So this one is different. This is when the task is performed, but the item is actually not checked off.
Leah Kulp: This was 7% of all possible checks were failed checks. The most common ones were for checking the patient's temperature on the primary survey.
Leah Kulp: And making sure that the oxygen mask was connected and giving food or blood are qualitative observations showed that
Leah Kulp: Leaders missed virtualization of temperature due to the noisy environment. So when we went back and check the videos we often found that the provider, one of the nurses was
Leah Kulp: Reporting the temperature. But there's a lot going on and the leader with the checklist just missed this exam.
Leah Kulp: Second, the giving fluid and blood does not apply to all patients actually the fluid and blood item is automatically
Leah Kulp: The na not applicable checkbox is checked off. And so if fluid or blood is being administered to patients, the leader would need to uncheck the NA box and then check off the corresponding box.
Leah Kulp: And finally, the pre arrival pre arrival items.
Leah Kulp: Failed to check if item. If the team leader arrives late. So this means that when the team leader arrives to the room after the patient.
Leah Kulp: Maybe even once the team has started the primary survey, they'll have missed all of the pre arrival tasks. So even though these tasks were completed the leader might not check off these items because they weren't in the room when they were done.
Leah Kulp: Next, I'm going to talk about how the timing of check offs related to the task performance. So this was the comparison of timestamps between activity logs and checklist logs.
Leah Kulp: First we categorize all of the true checks as check before the test started during task performance or after the task was completed.
Leah Kulp: We found that 15% of all these checkups were pre checks. So this occurred before the tasks even started. The most common ones or the heart rate, the back exam and the warm blanket placement
Leah Kulp: So we noticed that these tasks were tasks that were long duration. So they might take a little bit longer or multi step tasks.
Leah Kulp: Secondly, we looked at check off that occurred during task performance. So these we called real time checks and this was 16% of all true checks. The most common ones were the C spine exam. The back exam and confirming that oxygen mask was placed on the patient.
Leah Kulp: Again, our qualitative observation was that these were long duration tasks was easier to check off during past performance.
Leah Kulp: Finally, we looked at tasks that were checked after task performance. We call these post checks.
Leah Kulp: 69% of these post checks and this is what we would expect. Right. The task is performed and then the leader diligently checks the box off.
Leah Kulp: The most common ones were for vitals secondary survey items and then the pre arrival items. Some of the opposite observations were that the vitals actually show up on a monitor, once the sensors are connected to the patient. And so the leader can just quickly check these off.
Leah Kulp: And we also found that the pre arrival items checked after the leader arrives. So some of these tasks are done and then when the leader arrives, they're checking things off. So it's checking off after these tasks were performed
Leah Kulp: As I mentioned, we would expect most checkoff to be post checks pastors performed item is checked off.
Leah Kulp: However, what if these check offs are very delete. We also consider this non compliant behavior, there's a gap of, say, five or 10 minutes between when the task performed and when the item was checked off, then this checkoff doesn't really accurately reflect reflect a task performance.
Leah Kulp: So to determine the threshold for when the post checks are delayed we calculated the time difference between the median of task completion. So from start to end and the median of
Leah Kulp: The item one, item was checked off for all 98 cases, we found that two minutes or longer was our threshold for a delay so task is performed. Two Minutes Later team leader is checking the box off.
Leah Kulp: And the items with the moat longest delays were mostly pre arrival items, as I mentioned, these tasks might be done before the leader even arrives to the room and then the vitals, which are showing up on the monitor. And finally, warm blanket placement, which occurs during the primary survey.
Leah Kulp: We wanted to take a closer look at what context was affecting these delays.
Leah Kulp: And while these delays might not seem like a long time, only two minutes. They could leave some negative outcomes and concurrent checklist use because, as I mentioned, it does not accurately reflect check task performance.
Leah Kulp: So next I'm going to talk about the association between those delayed check offs. And then the context attributes.
Leah Kulp: First, we had to split up all of the checklist items into different sections to reduce the amount of analyses that we were doing.
Leah Kulp: So we looked at overall the checklist so pre arrival primary, secondary, and then the prepare for travel section. And then we also looked at only the primary and only the secondary surveys
Leah Kulp: And the context that we found were associated with delayed check offs included the team leaders late arrival, as I mentioned this affected all sections on the checklist. So when the team leader arrives late
Leah Kulp: The checklist items are more likely to be delayed after task performance. We also found that overall every section and then the secondary survey items were affected by patients with penetrating injuries.
Leah Kulp: So this shows that penetrating injuries and patients with more severe injuries are more associated with delayed checks.
Leah Kulp: And then finally, similar to the last finding secondary survey items were affected by attending level activations so this means the highest level acuity patient, there might be more providers in the room tasks are more delayed and item check offs are delayed.
Leah Kulp: I want to quickly discuss the factors contributing to noncompliant use and then how we can reconsider checklist designed to better support these contacts.
Leah Kulp: So first the work practices and task perceptions have formed at the bedside over time may not be the same as our coding dictionary.
Leah Kulp: So ACTUAL ORDER OF items or tasks that are performed really varies based on patient status, the team members that are in the room and other emerging context information.
Leah Kulp: Teams might not always follow the prescribed order of tasks, especially if there's an intervention, they have to place a chest tube or to beat the patient.
Leah Kulp: And these activities are running continuously they stop and resume and overlap. So it might be hard for the team leader to accurately mark when tasks are occurring based on the simple checklist.
Leah Kulp: And then second, the variable nature of task length and complexity.
Leah Kulp: Really contributes to checklist compliance. So as we saw tasked with longer duration or multiple steps were affecting non compliant behaviors. So now, how can we reconsider checklist designed to support these different types of tasks.
Leah Kulp: When work reconsidering checklist design a few things that we can look at are different types of tasks performance.
Leah Kulp: Supporting retrospective checking so checking after tests have occurred and then supporting timely documentation.
Leah Kulp: First, I'll talk about supporting different types of tasks performance and interest of time, I'm only going to talk about few of these.
Leah Kulp: So first, we found that tasks with long durations affected checklist compliance so design solution could be to indicate when the task has started and then later. Once the task is completed, indicating that completion.
Leah Kulp: This way the team leader can accurate more accurately document the task performance.
Leah Kulp: Second tasks that are performed more than once. So we did find that the vitals were are being reported multiple times and this feature would automatically pull from the vitals monitor and allow the leader to record multiple vital signs values.
Leah Kulp: And third tasks with multiple steps so similar to the first design solution, allowing for multiple
Leah Kulp: Steps to be recorded individually, instead of just checking one box. Now, the leader can more accurately document this task performance.
Leah Kulp: So now supporting retrospective checking. So these are often tasks that are performed before the patient arrives or before the leader arrives.
Leah Kulp: As we mentioned when the leader arrives late the team has already performed lots of tasks.
Leah Kulp: And then they show up and are checking things off and they might be checked off with delayed or they might not be checked off at all. So how can we better support this.
Leah Kulp: The first option is a section for task performed before the main event. So this would be a pre hospital forum on the right is an example of what this might look like.
Leah Kulp: Instead of having the leader write these notes in giving a structured form would help streamline as data entry.
Leah Kulp: And also allowing other providers in the room to fill this out before the leader arrives, so that the documentation is more accurate to the test performance.
Leah Kulp: And being able to indicate whether the task was performed before or after the main event will also help leaders more accurately document. For example, the ID access is often done as the patient is coming to the hospital.
Leah Kulp: And when the team leader checks assign them off that might indicate that it's happening during the resuscitation, but it actually occurred before the patient arrive to the hospital.
Leah Kulp: So providing a way for leaders to indicate whether task will perform pre hospital would allow them to more accurately document what's going on.
Leah Kulp: And finally, supporting timely documentation.
Leah Kulp: We found that the vitals and the pre arrival items are checked off two minutes after task completion.
Leah Kulp: And from the video review, we found that some leaders just checked items off at the end of the case, instead of checking each item off after the task was performed
Leah Kulp: One way we can better support this is to show values that were previously entered. So with this pre hospital form, we can then show these values later when the team leader gets to tasks that are relevant.
Leah Kulp: Second, we can have a timer showing the last time since the first check off and this would help leaders.
Leah Kulp: conceptualize how much time since the patient has arrived in the room. And finally, showing timestamps for individual checklist items would also provide a better idea of the progress of the resuscitation for the leader.
Leah Kulp: So in conclusion, we are understanding factors that contribute to compliance and timeliness of checklist interactions and from these findings such as false checks.
Leah Kulp: Inaccurate checks and failed checks of non compliant behaviors. We can then reconsider checklist design, how to better support these non compliant behaviors to try to achieve 100% compliance with the checklist.
Leah Kulp: Thank you so much for your time. I'd like to thank my collaborators at Drexel University Children's National Medical Center.
Leah Kulp: And the funding bodies that have made this research possible. If you have any questions, feel free to email me LJ k at drexel.edu and I'd be happy to talk about it. Thank you.