Hi, I'm Christina and I will intoduce our paper
"Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality"
We wanted to develop an inside-out object tracking system working on an Augmented Reality HMD, such as the HoloLens.
Why do we want to do that? Our area of application is AR supported surgical navigation.
Surgical navigation systems aid surgeons during interventions,
by tracking medical tools in relation to the patient’s anatomy.
With conventional systems, however, surgeons have to divide their attention
between navigation information on a 2D monitor, and the patient.
AR HMDs can display the navigation information directly in the surgeon’s view of the operating site.
Thus, the surgeon can focus on the patient only.
Compared to conventional navigation systems, AR HMDs are inexpensive, and have a slim form factor.
This allows an implementation also outside of the operating room, opening up new areas of application.
Now, I’m sure many of you are familiar with Inside-out tracking libraries for image fiducials,
such as ArUco and Vuforia. So, could we just use these for tracking our tools?
Well, it has been shown that such of-the-shelf trackers have errors of several millimeters up to centimeters,
which makes them unsuitable for many surgical applications.
Image markers are also sensitive to occlusions and different lighting conditions,
and they have to constantly face the camera for tracking. This, in practice, limits the trackable movements.
Our idea is to track reflective marker spheres instead, which are already commonly used in optical tracking systems,
in particular, also in the medical domain.
This is usually very robust and accurate.
The spheres are attached to the surgical tools and tracked outside-in with cameras placed around the operating site.
Instead of relying on those external tracking cameras, we want to directly track with the HMD,
to keep all the advantages of the HMD’s slim form factor.
This leads us to our contribution: We developed an AR system for inside-out marker sphere tracking
which does not rely on any stationary infrastructure. Instead, we just use the on-board sensors
of a state-of-the-art HMD, the HoloLens 2.
We show that our system is capable of real-time performance, 6DOF tracking and accuracy in the millimeter range.
To accomplish this, we want to compute the 6DOF pose of a marked surgical instrument with respect to the HoloLens.
The HoloLens has four environmental tracking cameras. We use the two front facing ones, left and right,
and re-purpose them as a stereo vision setup. The left and right cameras are, unfortunately, not synchronized,
which is an important prerequisite for stereo vision. Consequently, we implemented a
single-constraint-at-a-time Extended Kalman filter, or short, SCAAT, for tracking with unsynchronized frames.
We found that the SCAAT is also more accurate and faster than conventional stereo, and therefore,
use the stereo path only for initialization and reinitialization in case tracking through the SCAAT is lost.
To prepare for tracking, two offline steps need to be completed:
The first is stereo calibration of the two HoloLens cameras. We require intrinsics, as well as extrinsics,
so the transformation between left and right camera. This calibration has to be as accurate as possible –
We found the on-board calibration of the HoloLens to be not sufficient. Instead, we use a calibration framework
for low resolution cameras.
Second, we need to prepare the instrument we want to track, which is a rigid body defined by its marker spheres.
For pose estimation, at least four, better five, marker spheres with unique relative pairwise distances
are necessary, to be robust against occlusion and overlaps.
Now, we can move on to the online tracking steps. The first step there is the marker detection.
We filter and threshold the images obtained from left and right camera, and perform connected component search
to obtain a set of marker candidates, x prime.
To improve runtime, we perform some of these operations on the HoloLens GPU, and use a linear kalman filter to track
regions of interest around already identified markers, and apply the marker detection only to these regions.
If enough markers have been detected, and if the time difference between left and right frame is below 1 ms,
the frames are candidates for pose estimation through stereo vision. Here, we use a classical triangulation approach.
First, we match markers between left and right frames using the epipolar constraint.
Second, from the matched markers, we can now compute 3D points by using the optimal triangulation algorithm.
And third, we match the known point configuration of our instrument to the point cloud obtained from triangulation.
If the matching is successful, pose can be computed by solving the absolute orientation problem.
As already mentioned, we use the stereo vision pipeline only for initialization and re-initialization.
Otherwise, we track with the single constraint at a time extended kalman filter.
A SCAAT is a filter which estimates the state z of a non-linear system based on partial or incomplete
measurements x prime. In our case, we can process every frame we obtain, regardless of synchronization
and of how many markers have been detected in it. So by using the SCAAT, one frame with one measurement
is enough to update the pose estimation.
We describe the movement of the instrument with a position-velocity model.
So, the state z contains the current position and rotation of the instrument.
Then, for each frame with an incomplete measurement of markers x prime, the SCAAT first does a prediction,
so it predicts the state z hat for the current time point using the underlying non-linear motion model.
Then, the predicted state is back-projected to the current frame, yielding a predicted measurement x hat.
The predicted measurement x hat is now compared and matched to the actual measurements x prime
and the residual between the two measurements is computed.
And finally, the state of the filter is updated based on this residual.
The final, corrected state now describes the instrument‘s pose at the current time.
We performed a technical evaluation of our system. For all experiments, we compare the instrument poses computed
with our application with the equivalent pose obtained using a high-precision, optical tracking system.
First, we measure tracking accuracy in static and dynamic scenarios. We achieve an error
of 1.7 mm in translation, and 1.1 ° in rotation in static and of 1.9 mm and 1.2 ° in dynamic settings.
As shown, the tracking error is slightly dependent on the distance of the instrument from the HoloLens,
with the highest accuracy between 20 and 60 cm. Since, for our application,
a rather close operating range of instruments can be assumend, this is feasible.
It is evident that dynamic tracking is slightly less accurate than static tracking,
which is caused by an under-sampling of motion, and self-occlusions in the SCAAT-EKF.
We also compare our SCAAT-based tracking with our stereo tracking only,
and with tracking of a commonly used image fiducial, an ArUco marker.
The ArUco marker is tracked monoscopically with the video camera of the HoloLens.
Our SCAAT-based approach outperforms monoscopic image tracking by a large margin.
We also observed the other advantages of spherical marker tracking already mentioned before,
such as higher robustness to occlusion and different viewing angles.
The accuracy of the stereo only path is very strongly dependent on the instrument distance.
This is caused by hardware limitations enforced by the given stereo configuration within the HoloLens.
Tracking with SCAAT can mitigate this limitation, by implicitly filtering noise in movement and measurement,
and minimizing the re-projection error over time.
Furthermore, thanks to our efficient implementation, the SCAAT-EKF has the fastest runtime of only 1.5 ms per frame,
and therefore, easily achieves real-time performance.
To conclude, we presented a method for inside-out, six degrees of freedom tracking of marker spheres
using only the HoloLens 2 as hardware.
Our system can be seen as a drop-in replacement for applications which use optical outside-in object tracking.
As an example, we chose surgical navigation: With our system, a surgeon wearing the HoloLens
can use the familiar marked tools, but without requiring external tracking.
Our tracking system achieves real time performance, while approaching the reference precision
of conventional surgical navigation systems of below one millimeter positional, and one degree rotational.
With that, we think that it can already be used for interventions
where sub-millimeter preciseness is not required.
Thank you for your attention, and I‘m looking forward to answering your questions.