Hello, my name is Sunmin Son,
and I am here to introduce a novel tactile interaction that we call ThroughHand, on behalf of the KAIST HCI Lab.
This is a 2D pin display.
People with visual impairments use fingers to scan the 2D space and use buttons to manipulate the system state.
Pin displays require an interaction that consists of two phases: scanning and manipulating.
The scanning phase not only takes a long time, but also requires users to scan the display using their fingers repeatedly in order to perceive changes in specific parts of the display.
Therefore, multiple dynamic objects cannot be perceived simultaneously.
In addition, it is difficult for users to seamlessly transition between the manipulating phase and the scanning phase.
Since touch is reserved for scanning the tactile output, users are required to perform input using indirect buttons.
The two-phased structure causes difficulties in interaction with multiple objects.
We propose ThroughHand, which uses both the palm and back side of the non-dominant hand.
ThroughHand users perceive tactile output using the palm, which allows them to immediately sense changes within a large area.
Simultaneously, users are able to utilize the back of the hand as a touch input space.
Since the non-dominant hand is used both as an input channel and an output channel, we call this design co-located input and output.
We designed a 2D pin display of 12 by 17 pins with 10 mm of spacing.
We used an optical tracker for touch sensing.
We conducted a user study in a total of three steps with 10 visually impaired users.
The goal of the first experiment was to investigate the accuracy of touch interaction through the hand.
In the second experiment, we tested the performance of three basic interactions.
The basic interactions were then used to create several interactive applications, which were evaluated in the third experiment.
Today, I will summarize the first and third experiments. For the second one, please refer to our paper.
In the first experiment, we investigate how much additional error is made by performing touch inputs through the hand.
The subject performed a simple task: when a pin rose to stimulate the palm, the subject touched the corresponding location of the stimulus as accurately as possible on the back of the hand.
The average error was 13.2 mm. This is remarkably small considering that the two-point discrimination threshold of the palm is 10 mm.
Another interesting finding was that the top area (where the fingers were located) had significantly less error than the bottom area of the palm.
In the third user study, we implemented and tested four interactive applications.
In Shoot em up, the player manipulates an aircraft to shoot down threatening enemy planes while avoiding falling obstacles.
In Whack-A-Mole, the player selects randomly appearing moles as quickly as possible.
In Multi-player Pong, two players play against each other by controlling a paddle to hit the ball to the opponent's side.
In Rhythm game, the player selects falling targets that match the rhythm or notes of a song with the right timing.
After the game experience, we asked the subjects if the size of the palm was appropriate for playing games.
Seven participants answered that it was large enough to play the four games.
However, two participants responded that they could have better enjoyed the games if they had bigger hands.
For some users with acquired visual impairments, ThroughHand allowed them a similar experience of visual games that they played before they became visually impaired.
One user said that ThroughHand games were more enjoyable because the additional tactile output provided richer feedback when compared to games that only used auditory output.
Another user commented that ThroughHand was better because it had more operability and gameability.
One user said that ThroughHand’s Rhythm game was completely different from the audio-based rhythm game that he had played before.
It allowed him to concentrate on the music. He added that he wanted to play Rhythm game while listening to his favorite latest songs.
We designed ThroughHand based on the idea that touching the back of the non-dominant hand helps in fine-tuning the touch location through the feedback loop between tactile sense and finger movement.
However, we could not confirm this idea’s effects.
If proprioception without the feedback loop has high accuracy, we can use a touch sensing plane placed between the back side of the non-dominant hand and the fingers of the dominant hand.
Such touch input alternatives can reduce the size and price of the ThroughHand device to make it more suitable for popularization.
In summary, the user study confirmed the possibility of real-time 2D interaction "through" the hand.
Our ThroughHand games successfully provided a real-time 2D interaction through the tactile channel, along with an enjoyable game experience.