Hi, I’m Kai-Chieh.
I'll be presenting the first half of the presentation, then Chen-Kuo will finish the rest
Digital pens, a great companion for many designers, are becoming more and more versatile.
Not only can we write and draw with these devices on a screen,
recent developments show promising results for using these pens in midair.
And, in addition to being merely a pointing device,
digital pens can utilize its 3D orientation to convey information to the target device.
This is not all that digital pens have to offer, information can also flow from the pen to the user.
And haptic feedback is a great way to achieve this, since the user is almost always in contact with the pen.
Many researchers have studied how haptic feedback can enhance interactions with digital pens.
For example, Lee et al. studied the effect of generating a short vibration in the lateral direction of the pen
to simulate the sensation of clicking on a physical button
(something similar to what we would feel when touching the home button of an iPhone 7, which is not a physical button).
There are also work on simulating the vibrational sensation of writing on paper or other surfaces.
Some other work studied force feedback when the user interacts with items on the screen or in midair.
However, we realized there are a couple research directions amiss from previous works.
1. Although past research has shown that haptic feedback makes using pen as a pointing device more enjoyable,
no study has been published concerning its effect on the emerging scenarios where the orientation of the pen is used as input.
Providing haptic feedback as users tilt or rotate the stylus could potentially enhance these interactions.
2. Despite that many types of haptic feedback have been explored, a very fundamental and intrinsic characteristic of all pens remains unexplored:
the balance, one of the key contributor to how a pen feels in our hand.
You may have felt it was different writing with a capped and an uncapped pen; this has a lot to do with the change in balance of the pen.
Compared to other haptic feedback, such as vibrotactile feedback,
using the balance to generate haptic feedback is less disturbing to the user as it is an intrinsic property of all pens.
We may be able to be undisturbed by the fact that the balance of the pen feels different, but it is hard to be undisturbed by constant vibration.
Furthermore, weight shifting has been proven effective in giving users a perception of weight change in VR and other applications,
it would be interesting to know how it would work on a stylus.
Seeing these gaps in research, we set out fill the gaps, beginning with the design and creation of a prototype.
We named the prototype “Glissade”,
which means “to slide down a steep slope”, because its key mechanism is a brass cylinder sliding up and down the barrel.
A mini-gear motor mounted on the top of the pen controls the position of this brass cylinder by pulling on the toothed belt.
The exact position of the brass cylinder is calculated using data from a magnetic rotary encoder that precisely tracks the motor’s rotation.
And to track the orientation of the pen, we used an inertial measurement unit.
It gives us the yaw, pitch, and roll of the pen as well as the direction of gravity.
All digital components are controlled by an Arduino Mega board, which is connected to a computer.
Furthermore, we deployed a micro planetary gear motor, a 1.35-g component strong enough
to pull the brass cylinder against gravity at a decent speed.
The greatest challenges of making this stylus are its small diameter and light-weight requirement.
According to research, the optimal size of a pen is approximately 8mm in diameter.
And in order to create sufficient balance shifting without making the stylus overwhelmingly heavy,
all parts of the stylus needed to be chosen carefully.
Striving to meet these requirements, we designed the prototype around a 10mm wide acrylic tube.
To fit a belt in this tube, we cut a wider belt into strips of 1.5mm.
We also printed our own gears to fit inside the barrel and we made a custom wire bundle to reduce weight.
Once the prototype was complete,
we designed 6 ways to move or position the mass inside Glissade to generate what we call the balance shifting feedback.
We call them: balance patterns.
Remember that one of the goals for this study is to explore the effects of haptic feedback when the user tilts or rotates the stylus as a means of input.
Therefore, we designed three dynamic balance patterns,
for which the mass changes position according to either how much the user tilts or rotates the pen.
The first pattern is called shift towards the front:
meaning that the mass shifts towards the front of Glissade as its orientation deviates from a reference point.
The opposite of the shift towards the front pattern is the shift towards the back pattern.
Then there is the balance oscillation pattern.
The mass slightly and briefly moves up and down once
whenever the pen orientation crosses a certain angle with respect to the reference orientation.
The other three patterns are static, meaning that the mass doesn’t move when the user changes the orientation of the pen.
Hi, I’m Chen-Kuo Sun let me continue to present.
We conducted a study to test the balance patterns we designed,
with an aim to answer the following questions :
How well can they distinguish different static and dynamic balance patterns?
Would the recognition rates be affected by the way the pen was used,
the pen tip’s contact with a surface, or extra cognitive load?
So, we recruited 12 participants. For this recognition study.
Compared to other auxiliary input channels, such as pressure or rolling,
tilt and rotation involve more pen motion,
which induces more perceivable inertial force from the mass inside Glissade.
Thus, we examine these two parameters in two identical experiments.
During each trial, participants performed tasks under one of the Workspace 、 Secondary Task and Pattern combinations.
The tests were done in blocks.
In each block, consists of 48 trials as a result of the six balance patterns being presented in random order eight times.
For tilt and rotation , there are 4608 trials for the two experiments.
The Workspace parameter examines whether or not the normal force from a surface in contact with the pen affects balance pattern recognition rates.
In With Cognitive Load condition under Secondary Task,
participants were asked to divert some of their attention to a secondary task,
simulating scenarios in which some attention is diverted to other visual contents.
The participants were allowed to familiarize themselves with the patterns before commencing the experiments.
Participants were asked to tilt or rotate the pen to 45 degrees.
Both pen motions were completed in a single stoke.
After a participant had completed a stroke, the experimenter would take Glissade out of the participant’s hand,
and the participant would either request a retry or call out the name of the pattern.
So, the followings are the results.
By allowing retries, we acquire the absolute ability for users to distinguish patterns and their confidence in their response.
In the Pen Tilting Experiment, average recognition rate across all conditions is 94.27%.
The results of ANOVA analyses show that, there is no significant three-way interaction between Workspace,Secondary Task, and Pattern.
Only Pattern has a significant effect on the recognition rates,
we discovered that the recognition rates of the dynamic balance patterns are significantly higher than those of the static balance patterns.
The subjective feedback from the participants and the confusion matrices indicate that
this resulted from the difficulty participants experience when differentiating between Front Heavy and Balanced pattern.
No significant differences in recognition rates are found between the dynamic balance patterns we selected
In the Pen Rotating Experiment, average recognition rate across all conditions is 94.23%.
Similar to the experiment aforementioned, there is no significant three-way interaction between Workspace,Secondary Task, and Pattern.
Pattern has a considerable impact on recognition rates,
dynamic balance patterns are all significantly higher than static balance patterns.
The results of the confusion matrices regarding the static and dynamic patterns in this experiment are similar to previous experiments.
The participants reported that, when performing the input action at a fast speed,
Front Heavy and Balanced sometimes create similar sensations of weight.
However, they agreed that if they manipulate the pen at a normal speed, the difference between the two patterns can still be recognized.
Also, there were three participants who reported that their performance on Front Heavy and Balanced patterns
could be increased if they could "use the pen daily to get more familiar with the balance perceptions.
The results of the two experiments suggest that participants were able to
identify balance shifting patterns for on-surface and off-surface scenarios.
Even if the participants were engaged in parallel tasks, the patterns could be distinguished within a few attempts.
We illustrate a few new interactions made possible by the application of balance shifting feedback on digital pens.
We propose using Glissade to create a haptic editing board, a user can select a button to open a pie menu,
and the pen displays the balanced pattern, This helps the user understand that the menu mode is activated.
In total, the pie menu contains four commands, copy, paste, delete, and adjustment.
Where the user can sense static and dynamic patterns during auxiliary pen input.
Glissade also enables new interactions in gaming for surface computing.
In a shooting game, the balance can simulate the amount of ammunition left in a gun.
The balance becomes slightly more front heavy whenever a shot is made in the game.
In a bowling game, for example, a more back heavy balance simulates a heavier ball.
Furthermore, in a game where the user bowls by swinging the pen,
shifting the moving mass downwards with the swing creates a haptic sense of throwing the ball.
Aside from expressing the current state of the pen,
static balance patterns can be used to convey long-duration system information.
For example, when reading a document, the balance of the pens can express unread messages to avoid interference by the message window.
In addition, we have designed a countdown timer that can operate in the background
so that the user can realize the time and focus on an application.
In summary, balance haptic feedback enhances interaction enjoyment.
The rating enhancement was greatest in the two games,
because participants felt that the feedback makes them fun and is especially suitable for expressing weight.
Here we recognize the limitation of glissade.
Our study focused on simple balance patterns to demonstrate the promise of Glissade.
Future research will go beyond the discriminability of the balance patterns by investigating deeper questions
regarding the perceivability and discrimination threshold values of the kinesthetic feedback.
Such as Just Noticeable Difference experiment.
To understand the maximum number of the static patterns and other possible combinations
of upward and downward mass movements for dynamic patterns.
The current Glissade prototype is connected to an external power source and a controller via a wire bundle.
To further enhance Glissade’s mobility,
the wire can be removed by integrating into the pen a small battery and a miniature controller with wireless capabilities.
Finally, we explored coupling balance shifting feedback with the orientation of Glissade:
an ungrounded digital pen prototyped to demonstrate the potential of the novel haptic feedback.
Six balance patterns, including static and dynamic balance patterns.
And the results show that the participants were able to recognize balance patterns with a 94.25% accuracy.
Also, we found participants appreciated our applications.
We believe our initial investigations in balance shifting feedback open up a wide range of possibilities
for not only auxiliary pen input but also a wide range of potential applications for surface computing.
Thank you for listening