Hello, I am Danilo Gasques from UC San Diego, and I’ll be giving an overview of our work ARTEMIS
a collaborative mixed-reality system for immersive surgical telementoring
When someone suffers a traumatic physical injury, they usually require immediate
medical attention. In some situations, such as in a natural disaster, the right medical expertise
might not be readily available. To save the patient, local surgeons rely on the help of
remote experts through telementoring systems. In a typical telementoring system
the expert surgeon guides novices verbally and through video annotations. Unfortunately, telementoring
forces the novice surgeon to shift focus away from the patient to the telementoring screen and then back
which sometimes leads to procedural mistakes. (Telementoring) also fails to incorporate the physical
nature of surgery as remote experts can't demonstrate the use of specialized trauma tools
Researchers have been addressing limitations of current telementoring systems by using
mixed reality displays. Mixed reality displays are displays that sit between
the user and the environment. They can show visual guidance situated on the patient
which helps, for example, in overcoming the focus shift problem we discussed before.
Today, mixed reality research on telementoring is mostly focused on the novice surgeon
remote experts still make use of regular flat screens. This gives us a lot of opportunities
to improve communication between novice and expert by also designing with the expert in mind
The literature of mixed reality in remote collaboration is rich with examples of visual
affordances that can improve communication between remote parties. For example, showing
representations of remote objects, virtual hands, avatars, and more. So, to better understand how
this rich literature applies to the domain of telementoring, we conducted formative
study. We invited the first surgeons to role-play surgical procedures in a mock-up operating room
during the sessions surgeons mentored one of our researchers on two common trauma procedures
first, side by side; and, then by using existing mixed reality applications. The second part allows
us to understand how different affordances of existing mixed reality technology can influence
their ability to mentor novice surgeons. At the end, we video recorded the sessions and analyzed
them to observe how expert mentor novices the amount of information they need as well as what
they showed to novices. We summarized our findings from the role-playing sessions into four goals
to describe what expert surgeons need to better mentor novice surgeons. I recommend you to take a
look at the paper for an in-depth discussion; but in a quick comparison with existing literature
expert goal (EG) 2 requires them to mark not only the location and length of the incisions but
also their depth. With these goals in mind, we designed Artemis a telementoring system that combines virtual
reality for the expert and augmented reality for the novice. Let's go over each interface starting
with the expert's interface first. The expert interface is in virtual reality but instead of
using bulky virtual reality controllers, experts use their hands to, for example, enact the use of
surgical tools. Experts can also use a customized pen to make annotations on the patient. They see
the patient in 3d as a live reconstruction, and they can use this patient to show the location of
anatomical landmarks as well as mark the location, length, and depth of incisions as seen in the video.
Experts have access to several camera views into the operating room
this allows them to, for example, look at a procedure from a viewpoint free of occlusions
or from a camera on the novice surgeon's head
the interface is a mini surgical room, so to better understand where the novice is
the expert surgeon can see the avatar of the novice
finally the novice interface is simpler as it uses augmented reality to show an novices both
the patient and the visual guidance at the same time. For an already overwhelmed novice surgeon
the interface is a passive display of visual information controlled by the expert surgeon .
We conducted a qualitative evaluation of our system with six novice surgeons performing
a range of emergency procedures on manniquins as well as cadavers
overall, ARTEMIS addresses the proposed expert goals as well as it enables
them to communicate with novices in ways that are impossible to do even in real life for example
in this video the expert uses their hand to show a way the novice can find the incision
point on the manniquin by feeling its larynx. Thank you for watching and don't forget to
read the paper for more details on the system implementation as well as on the discussion!