We propose a simple and natural data collection method called ‘styling words’ to improve gesture recognition performance
Gestures have become a more common tool for human-computer interaction. Because of its own intuitiveness and usefulness
But, human gestures are easily affected by various factors, such as affective or physical body conditions
So gesture recognition still remains one of the most challenging problems in computer vision
It is the common way that uses a machine learning model with a large-scale dataset to improve gesture recognition accuracy
However, it takes high human resources and time to build a large-scale dataset. To solve the problem, we gave attention to the variability, one of the features of it
We propose a word-based instruction method called ‘styling word’, that can induce enough and proper gesture variability even though you build a small-scale dataset
To prove the effectiveness of ‘styling words’, we chose 12 “Sondam” gestures, a visual word dictionary recently developed to facilitate interaction between individuals having severe developmental disabilities in South Korea
And We designed eight intuitive and abstract styling words to induce gesture variability
This slide shows the gesture videos without styling words Keep in mind and let’s look at the next slide
When the Styling words are applied to gesture collection, it is easy to find out the gestures are changed
And in order to quantitatively measure the change in variability, we extracted the human skeleton by the 'openpose' model
With the extracted skeleton, we tried to analyze it with specific measures. From the study of Vatavu, we could get some useful features for the gestures
However, the features are not perfectly fit our data, because of dimension, we redefined them and it is written on the slide
In the right figure, we show the differences among gestures with Styling word type A, Styling word type B, and without Styling word
Each vertex of the hexagon represents gesture area, quantity of movement, performance time, average speed, body posture variation, body posture ratio
As you can see, for a participant, features of gesture with styling words have much bigger variability comparing to without Styling words
Unlike what we expected, type A showed more noticeable variability than type B
The degree of difference depends on the participant and gesture, however, they show the difference on average
We evaluated the performance of the with and without Styling Word model
The evaluation was basically performed using the leave-one-out cross validation, and the number of participants was arranged from 1 to 20
The test result of with Styling Word model was 9.3% higher based on 20 people
In addition, as can be seen from the dotted line in the left graph, the accuracy of without Styling Word model of 20 people and with Styling Word model of 4 people were similar
This means that the Styling Word method can be a great help when reducing the number of participants
And the box plot shows the results of each cross-validation
This also confirms that the generalization performance of the with Styling Word model is more stable
Finally, the right graph is the recognition performance by styling word type
Type A consists of more intuitive words, such as “fast" and “huge", and type B consists of abstract words such as "elegantly" and "roughly"
Before the experiment, we expected abstract words would affect recognition accuracy more than intuitive ones
But, As shown in the graph, intuitive words have higher recognition performance compared to abstract words
This study introduced a more efficient methodology of collecting human gesture data by supplementing directions with words to increase variability, even when restricted by small sample size
And, styling words also have the potential to be applied to the data collection for extensive modalities such as voice, touch, and so on
Thank you for watching and have a good time with CHI conference