Hello, I'm Dongjae Kwon, a Ph.D. student at the Artificial Intelligence Graduate School at GIST, South Korea.
Our team has investigated the validity of crowdsourced experiments for Tacton evaluation in both pairwise similarity and metaphor ratings.
I'm excited to share the results with you.
Nowadays, tactile displays have become a promising technology for transmitting information.
Tacton, a composite term of "tactile" and "icon," is a well-known concept for conveying information with tactile signals.
Many applications, including VR controllers, mobile devices, and haptic suits, have utilized Tactons to maximize their value in end-user applications.
To design effective Tactons, haptic designers usually collect pairwise dissimilarities and analyze the perceptual space of Tactons to find the most distinguishable subset.
Designers often run a controlled lab experiments to obtain reliable and robust dissimilarities by controlling the haptic hardware, user’s attention, posture, and other environmental factors.
However, the lab study requires a face-to-face experiment which results in additional costs.
One recent alternative includes running user studies online with crowdsourcing platforms.
A few studies have crowdsourced haptic data collections using mobile devices, especially for affective or subjective ratings.
The results were promising, but they focused on a small subset of Tactons that varied vibration parameters systematically.
Therefore, the crowdsourced haptic studies requires the validation on a variety of Tactons.
To investigate the validity, we evaluated the consistency of responses between lab and crowdsourced settings by comparing pairwise similarity ratings and metaphor ratings for two different Tacton sets.
We had two research questions.
First, can crowdsourced experiments achieve consistent results with the lab setting for perceptual similarities for Tacton sets?
Second, can crowdsourcing provide reliable results for assessing the subjective metaphor rating?
To answer these questions, we designed two user studies.
Pairwise similarity rating task is performed in User study 1 and 2, and metaphor rating task is performed only in user study 2.
And we developed an iOS application for the two rating tasks.
Our app consisted of four sessions.
First, participants input their demographic data in initial session.
Second, they were asked to feel all the vibration Tactons at least once in training session.
Third, participants gave similarity ratings for each pair of Tactons after feeling both vibrations.
Finally, they finished the experiment by providing any comments in the feedback session.
In user study 1, we selected 12 Tactons by varying three parameters including carrier frequency, duration and envelop frequency.
These parameters are used for affective rating task in previous work, but they did not assess pairwise similarities.
We measured the similarity ratings for all pairs of the 12 Tactons with 67 trials.
And we added on attention test pair consisting of an identical pattern, to check whether the participant focused on the experiment or not.
Using these vibrotactile Tactons, we ran two experiment in crowdsourced and lab settings.
For the crowdsourced setting, 20 participants were recruited.
Participants were paid $3 dollars, and the experiment took about 15 minutes.
Various iOS phones were used as you can see on the right side.
For the lab setting, 20 participants were recruited as well.
Participants were paid $11.5 dollars, and the experiment took about 15 minutes.
All the participants used the same iPhone 11 Pro Max.
They were asked to listen to the pink noise from the noise cancelling headphones during the lab experiment.
To compare pairwise similarity rating from lab and crowdsourced in User Study 1, we used Spearman’s Rank Correlation coefficient and Multi-dimensional Scaling analysis.
Spearman’s Rank correlation value was 0.93, showing strong consistency between lab and crowdsourced results.
From MDS, we derived 3D perceptual spaces and plotted them in 2D plots as shown in this Figure.
As you can see in the figures, the three vibrotactile parameters were aligned with the three dimensions clearly.
In User Study 2, we sampled 14 Tactons from VibViz library that includes 120 Tactons with user-annotated metaphors.
From the VibViz library, we selected 7 metaphors with minimal overlap between the different metaphors.
The 7 metaphors are listed on the left side. They include Alarm, Animal, Engine, Heartbeat, Jumping, Tapping and Walking.
The temporal plots for the 14 Tactons are shown on the right.
Using these metaphor-based Tactons, we ran two experiment in crowdsourced and lab settings.
For the crowdsourced setting, 20 participants were recruited.
Participants were paid $4.5 dollars, and the experiment took about 40 minutes.
Various iOS mobile phones were used as you can see in the right side.
For lab setting, 20 participants were recruited too.
Participants were paid $15 dollars, and the experiment took about 25 minutes.
All the participants used the same iPhone 11 Pro Max.
Participants were asked to listen to the pink noise from the noise cancelling headphones during lab experiment.
The pairwise ratings in User Study 2 resulted in a Spearman’s Rank correlation value of 0.91 between lab and crowdsourced data.
These results demonstrate strong consistency between the two settings as in User Study 1.
MDS analysis of the perceptual space suggested 4 dimensional coordinates.
We present the results in 2D plots as shown in Figure.
Along the first two dimensions, the spatial configuration of Tactons was largely consistent between the two settings as you can see in the highlighted sample points.
Our second research question was: can metaphor rating have consistent results between lab and crowdsourced settings?
For answering this question, we used equivalence testing to assess whether the metaphor ratings are statistically equivalent in the two settings.
The results showed that the ratings were statistically equivalent for 60%, different in 10%, and uncertain in 30% of lab and crowdsourced comparison.
These results implied that metaphor ratings may require a study design with more control for participants' demographics and background.
Now, I’d like to sum up my presentation with the following observations.
First, pairwise-similarity ratings showed high consistency between lab and crowdsourced settings for both systematic and metaphoric Tactons.
Therefore, perceptual similarity rating tasks can provide valid results in crowdsourced experiments.
Second, metaphor ratings can depend on many factors such as users’ demographics and background.
Therefore, cognitive tasks may require a more controlled study design to obtain consistent results in lab and crowdsourced settings.
Thanks for listening to my talk. If you have any questions, I would be happy to answer them at the conference.