Hello, we will present the method for automatically selecting a rejection threshold for custom gestures, which we call the Voight-Kampff Machine or VKM for short.
Gestures are human motions that intentionally communicate information.
We perform gestures with use of hardware, which captures and processes such input.
In the context of HCI, we use gestures to command software.
To facilitate end users and researchers, a lot of emphasis has been placed on gesture customization,
whereby a user can specify a gesture by example,
and after one or two demonstrations of a new gesture, the system can recognize the motion pattern, and that pattern can then be used to command system.
Gesture recognition using nearest neighbor pattern matching has seen significant progress.
These methods use straightforward similarity measuring techniques to classify candidate gestures.
That is, given a set of training samples and a query gesture, one can compare the query to each training sample
by measuring geometric similarities and select that which best matches the query.
This works well when we assume that the query must belong to one of the classes.
But what about when the query is not a gesture?
This is common with continuous input, where for each new input device sample we collect, we must decide if the current input represents a gesture.
This requires a rejection threshold such that when we compare a candidate gesture to a training sample,
if the similarity score is poor, above a certain threshold, we know to reject the input.
But without knowing the distribution of gesture patterns, because we only have one or two samples, it is quite challenging to select a robust threshold.
Our solution is called VKM. A common measure of accuracy and continuous gesture recognition is the F1 score, which balances precision and recall.
Our approach works to maximize the F1 score by leveraging synthetic data generation. VKM is a five step process that works as follows.
One, for each training sample, generate a distribution of within-class synthetic samples using a technique called GPSR.
Score each synthetic variant against training sample to generate a distribution of positive scores.
Two, for each training sample, generate a distribution of negative synthetic samples using a technique called mincer,
score each synthetic variants against the original sample to generate a negative score distribution.
Three, combine the distributions into a common positive and the common negative distribution from these find the threshold that maximizes F1.
Four, inflate threshold based on expected the variability.
Five, reduce the threshold based on the training set size.
First, gesture path stochastic resampling is a technique we previously developed to generate synthetic variants from 2D input trajectories.
For VKM, we extended this to 3D data. The details are which can be found in our paper.
Second, we introduced mincer, which generates a new negative sample by swapping out a random sub portion of its trajectory with another training sample.
This leads to much tighter boundaries using the VKM process described previously.
Third, using the scores generated in the two previous steps, we form a common distribution and from there
we select a threshold that best separates positive and negative scores to maximize F1 accuracy.
Four, gesture variability is application specific.
For example, gestures produced in games are more variable than those using standard data collection protocols.
One can inflate the threshold to account for this difference, assuming the training samples were collected in the less variable inducing environment.
Last, when we think of the threshold as a radius that generates a sphere around each training sample,
any points that fall in the sphere will be assigned to training samples class.
As we have more and more training samples, we can reduce the threshold and thus radius in order to reduce false positives.
When we only have one sample, we need a large radius.
But when we have several samples, we can get away with a small radius.
We can use simulation to determine what is an appropriate reduction. The exact details again can be found in our paper.
We equipped a continuous custom gesture recognition system with our rejection threshold selection algorithm,
as well as several alternative techniques in order to evaluate VKM.
We refer to this recognition pipeline as the Dollar General or TDG.
Machete and Jackknife are two important components of the pipeline.
Machete is a custom gesture segmenter, which ,for each time step, returns a likely start time of the current gesture.
Jackknife is robust recognizer, which then compares the segment's trajectory, against the training templates.
Using this pipeline, we collected user dependent F1 score performance data for four challenging high activity data streams,
that interleave gesture and non gesture actions where the system is trained with a varying number of training samples.
We compare it to VKM that uses mincer against six alternative threshold selection approaches based on prior work and intuition.
For example, three-sigma selects the threshold at three standard deviations above the combined positive scores mean,
and three-sigma-per-class does the same on a per-class basis.
The optimal technique is a theoretically highest achievable score that was predetermined by iterating over a range of all possible thresholds.
We use it as a baseline since it cannot be achieved in practice.
Using a follow the leader protocol, we collected the continuous Kinect, Mouse, Vive Position and Vive Orientation data for a total of four datasets.
We also collected five training samples of each gesture from each participant; 10 per input device.
So, the rejection threshold selection method had a significant effect on overall Kinect accuracy.
Mincer achieved near optimal performance across all train count levels and accuracy improves as the train count increases.
Three-sigma and three-sigma per-class on the other hand, did not perform as well and their accuracy decreases with increased number of training samples.
For Vive Position, several methods achieved near optimum performance with mincer outperforming other methods only at T equals 1.
All methods benefit from the increased number of training samples and the differences were not statistically significant.
The selection method had a significant effect on the Vive Quaternion accuracy, mincer and three-sigma-per-class both performs close to optimal.
Overall accuracy across all methods are lower for this datatype than on other datasets.
For Mouse, mincer and optimal achieve high accuracy above 90%, with only a single training template loaded,
the effect of the rejection threshold selection method on accuracy was statistically significant.
Three-sigma and three-sigma-per-class methods did not achieve high accuracy on any of the training counts, and decreased in accuracy with increased train count.
We also ran TDG on the SHREC 2019 competition gesture data, which consists of 13 participants gestures recorded with a Leap Motion device while the participants were an Oculus Rift.
VKM outperformed all methods reported in the competition results and was the only method to achieve above 90% accuracy.
This was possible only because of VKM's ability to select the tight rejection threshold to minimize the number of false positives.
TDG outperformed all competitors in the SHREC'19 competition with only uDeepGRU coming close with its level of performance.
UDeepGRU took over one hour to train while our C++ based TDG code takes less than 30 seconds to train on a MacBook Pro with a 2.3 gigahertz i5 core processor.
This is relevant in the context of customization where training must happen during runtime.
It's unclear how much of the currently working recognition pipeline can be used with new devices while the HCI community constantly prototypes and develops.
Whether optimal and equation for gpsr will work well with new types of 3D data is unknown.
Further, how well a single training sample describes the real underlying distribution of gesture variability is not known,
and it will be worth investigating online methods to analyze motion data on the fly.
To conclude, we presented VKM - a method for selecting an appropriate rejection threshold for custom gestures and high activity data.
Our approach outperformed all alternatives in our evaluations, is fast to train, and is device agnostic.
You can download a reference implementation and data on the ISUE lab's website.