I'm Tomohiro Sueishi from the University of Tokyo.
I'll make a presentation about "Ellipses Ring Marker for High-speed Finger Tracking."
Gesture recognition, especially finger tracking is a useful tool for interactive applications such as augmented reality and human-machine cooperation.
These videos of our laboratory show dynamic projection mapping onto deformable targets and high-speed interaction with a robot hand.
Fast and complicated hand-finger recognition will lead to more advanced interactions without latency discomfort,
but we currently suffer from a trade-off between latency and wearing continuity in finger tracking technologies.
Marker-less Leap Motion as a common hand tracking device has large latency due to high computational cost.
Marker-based motion capture systems can achieve low-latency tracking, but retroreflective dot markers are highly constraining, easy to drop.
So our purpose is to realize finger tracking with both low latency and high wearing continuity.
Many types of markers have been developed for spatial tracking, and commonly used markers are square-type and dot-type.
In case of finger tracking, we can equip regular polyhedron with the square AR marker on each side, but the fingertip attachment interferes with touch operation.
Dot cluster and circumferential markers can be designed for high-speed tracking, and especially the circumferential markers on a sphere of our previous work are effective against occlusion.
Ellipses ring marker, ERM is our new proposal, and uses elliptic features, a similar concept to this circumferential markers.
ERM has high wearability with finger-ring and retroreflective observable shape, and enables high-speed pose estimation using perspective invariance of ellipses.
ERM design are realized by the following procedure; cross-section of a cylinder with an inclined plane, central axis rotations, and combination of torus structures.
In this proposal, we adopt four symmetric ellipses for simplicity.
Actually, ERM has an advantage of surface normal distribution compared with a simple cylindrical side when we fabricate the ERM with retroreflective materials.
We simulated retroreflective appearance of both ERM and the simple cylindrical side. This simulation reflects retroreflective property of micro-beads type.
We can confirm the ERM can be brightly observable from various viewing angles, and this feature will benefit tracking of complicated hand motions.
I will explain the high-speed recognition algorithm of the ERM, core technique of our proposal.
This high-speed image processing is designed for high frame rate camera, and uses a self-windowing assumption; we only need to search the nearest regions. So this algorithm must run within every 2 ms.
First, we extract a region of interest, ROI around the ERM. We set the ROI position manually at initial frame, and after that as previous frame ERM center.
Then, we perform binarization within the ROI. Actually we currently uses a white marker with black background, but we assume the retroreflection can easily produce such high contrast image.
To eliminate noise area and thickness of ERM, we apply simplifying binarization and thinning operation.
And, we can get some nested contours as holes, and can roughly estimate the marker posture by solving a perspective-n-point problem.
We can calculate 2D intersections of ellipses by the estimated pose, and assign the detected contours to correct ellipses.
Finally, we can optimize the exact pose using the assigned ellipse features.
The optimization process of pose estimation uses the important fact that three-dimensional ellipses can always be observed as ellipses in perspective projection.
The 2D ellipse equation is expressed by six parameters, from A to F, and also expressed by a matrix Q.
We minimize re-projection errors of distances between algebraic ellipses and detected feature points by a least square method, Levenberg-Marquardt algorithm.
Unfortunately, we should be careful when the ellipses appear as straight lines; in this case, we cannot calculate properly the ellipse-point distance. There are two candidates, front and back.
Then we adopt a smooth weighting parameter lambda. This parameter can weaken the bad influence of straight line appearance.
We have experimentally validated the ERM performance using several devices.
We used 500-fps high-speed camera and infrared light. The observation were limited only in IR wavelength, and visible light was eliminated by some optical filters.
We also used high-speed monochrome projector for a visual feedback demonstration, dynamic projection mapping.
The ERM was fabricated by 3D printing of nylon. The radius was 10 mm, and we also fabricated dots marker on a simple cylindrical side for comparison.
Actually, the black sheet and materials were specialized for IR light absorption.
We evaluated tracking performance in three motion; pan rotation, tilt rotation, and finger free motion.
First, we could confirm that the computational time was sufficiently low, below 2 ms.
In several procedure of image processing, heavier tasks were thinning operation to eliminate marker thickness of binarization image, and iterative optimization using ellipse features, but these were also well below 1 ms.
As for estimated pose accuracy, we could confirm the pan rotation achieved sufficient accuracy similar to dots marker,
but in case of tilt rotation, we confirmed biased results of the ERM compared with dots. About 50 deg was estimated as 60 deg.
This bias will be caused by the use of only one side of the ellipse, typical phenomenon in conventional ellipse fitting.
Despite the bias, we could confirm sufficient performance of wide angle pose estimation and computational cost.
As a demonstration of high-speed finger tracking using the proposed ERM, we’ll show dynamic projection mapping onto a palm.
In a projector-camera system, a hot mirror acts to match these angle of view.
And we can easily realize the dynamic projection mapping only using a homography transformation assuming the palm to be a rigid and flat surface.
This demonstration is an example of high-speed low-latency real-time feedback to the finger.
Some research of a touch panel reports the perceivable latency threshold is about 6 ms.This demonstration can achieve such latency using 3 ms latency projector.
We can confirm almost no misalignment due to delay, and this will contribute to a palm-based display interface.
We have proposed an ellipses ring marker, ERM for fast low-latency finger tracking.
The ERM has advantages in low computational cost below 2 ms, and high wearing continuity as finger-ring shape and torus surface structure.
We also showed low-latency feedback demonstration, dynamic projection mapping. This ERM will contribute to low-latency gesture recognition for augmented reality and human-machine cooperative applications.
As future work, we will fabricate the marker with retroreflective material. This should improve usability.
We will also update several algorithms. Improvement of occlusion robustness, compensation of estimated rotation bias are necessary tasks.
Encoding modeling by asymmetric ellipses may also improve the usability of finger tracking.
Multiple ERMs with hand skeletal model are a further future task for several interactive applications.
Thank you for listening.