Hello, everyone.
I am Shigeo Yoshida, at the University of Tokyo in Japan.
I am glad to present PoCoPo,
a pin-based shape display that has the capability of holding itself for enhancing haptic experiences in virtual reality.
This work was done with Yuqian Sun,
and professor Hideaki Kuzuoka.
In this presentation,
we introduce PoCoPo,
the first handheld pin-based shape display that can render various 2.5D shapes in hand,
in realtime.
First, let me introduce our motivation.
In most VR applications,
it is quite common for users to touch virtual objects, hold them in their hands, and carry them around.
However,
it is difficult for users to believe that they are actually holding an object with the corresponding shape that they see in the virtual environment.
Reproduction of shape sensation in VR is still under development.
To make the VR experience more realistic,
researchers have developed several methods to provide haptic feedback to the hand.
However,
most of these methods focus simply on providing force feedback to the hands.
Since skin contact plays an important role in rendering shape sensations,
those methods cannot fully provide the sensation of holding objects with various shapes.
To challenge this issue,
we come up with PoCoPo that can render shape sensation in hand based on a pin-based shape display.
PoCoPo has 36 pins,
and each pin can be controlled independently similar to other pin-based shape displays.
Pin-based shape displays are a class of haptic devices
that can render the surfaces and shapes of objects using a set of thin pins.
As you can see here, many types of pin-based shape displays have been proposed.
Although these displays have a large display area and high resolution,
they tend to be heavy and, inevitably table-anchored.
Moreover,
since the actuators of existing pin-based shape displays are often placed underneath the pins,
they tend to be too big to grasp.
Some researchers have already proposed mobile shape displays.
Haptic Edge Display is a handheld shape display that can be attached to smartphones.
As Haptic Edge Display employs a single-line pin array,
the device can present only 1.5D shapes.
Furthermore,
output force of its pins is relatively weak,
so the device cannot maintain its shape while the user grabs it due to its backdrivable feature.
In contrast,
PoCoPo employs a non-backdrivable mechanism
to maintain its rendered shape while being held by a user.
Moreover,
PoCoPo has multiple pin arrays,
and this contributes PoCoPo to render 2.5D shapes.
PuPoP is another mobile shape display.
PuPoP renders shapes by inflating the small airbags holding in hand,
so it can render only pre-defined shapes.
To render different shapes,
PuPoP has to replace an airbag or attach multiple airbags.
On the other hand,
PoCoPo can change its shape instantly without changing its form factor.
Another limitation of PuPoP is that
it cannot simulate the continuous transformation of the shape of an object.
However,
our proposed device enables the dynamic transformation of the shape of the proxy
while being held by a user,
enabling the users to feel like they are holding a living creature.
Next, we will explain the implementation of PoCoPo.
Our challenge was how to make a pin-based shape display into a handheld size.
There are some kinds of actuators to move along a longitudinal direction.
A lead screw linear actuator might have enough power to push the user’s hand while grasping it,
but the mechanism tends to get larger along the motor direction.
A pneumatic actuator might be small
but requires an external air pump,
thus has less mobility.
Recently,
a reel-based linear actuator has been used for small shape-changing robots,
but insufficient power to maintain shape while the user grabs the device.
In PoCoPo,
we use worm-gear mechanism
and lead screw
for achieving its handheld size and non-backdrivable feature.
This is how worm gear of PoCoPo works.
The lead screw rotates with the worm wheel simultaneously.
The spacer with photomicrosensor is inserted under the worm wheel
so that it can read the rotation of the worm gear.
PoCoPo has a modular design
that facilitates the easy expansion of the rendering region,
depending on the purpose.
Four structures are put together for the one layer,
and we assemble three layers,
that is, twelve structures for one PoCoPo.
This is what it looks like when it’s actually assembled.
The pins protrude from two sides of the device.
One side touches the user’s palm base,
while the other side touches the user’s fingers.
Each side has 3 arrays of 6 pins;
thus, the device has 36 pins in total.
The display area, where the pins touch the user’s palm base and fingers, is
58 mm x 30 mm on each side.
The lengths of the pins were varied according to their positions
to prevent interference of the mechanisms.
Each pin can extend up to 3, 9, and 14 mm in the order from the outermost pin.
To track the rotation of the lead screw,
we embedded a self-made optical encoder with photomicrosensor
as a spacer at the base of the lead screw.
We manually drew a reflective pattern on the base with white and black markers.
Then, the optical encoder could read the changes in the color.
By assembling these elements altogether,
pins can move like this.
One of the obstacles in downsizing is electrical wiring.
Each pin of PoCoPo requires 3 wires for the photomicrosensor
and 2 wires for the motor.
As PoCoPo has 36 pins and requires 180 wires in total,
if you simply wire it up,
this is what happens.
This was the first working prototype of PoCoPo.
You can see PoCoPo, which was split into 4 parts now, here.
At this time, the circuit board was not directly attached to PoCoPo.
That's why all the wires were out of the device.
The cable was much heavier than the body of PoCoPo.
Then, by placing all of the circuits,
including motor drivers and multiplexers in a position
where they would not interfere with the user’s hand,
we successfully shortened the wire length.
In this manner, PoCoPo achieved its graspable size.
Next, let me introduce the user study.
We conducted two user studies to evaluate the effectiveness of PoCoPo.
In the first study,
we tested whether the participants could perceive the different shapes rendered by PoCoPo.
In the second study,
we investigated the acceptable range of the visual sizes of virtual objects
for a given shape rendered by PoCoPo and a given physical size.
Since we have limited time for the presentation today,
we only explain the user study for shape prediction here.
For study 2, please read our paper.
The purpose of the study 1 is
to investigate whether the shapes rendered by PoCoPo are distinguishable for the users by just holding the device without seeing it.
We covered the board to prevent the participant from seeing PoCoPo
and the rendered shapes directly.
To eliminate the effect of the way participants hold the device,
it was mounted on a bespoke stand,
and we permitted the participant to touch the device stably during the study.
A list of shapes was printed on a sheet of paper
and posted on a vertical partition board in front of the participant.
Following the previous studies,
we prepared 4 primitive shapes,
including rectangle, cylinder, sphere, and circle.
Moreover,
we would like to investigate
whether the difference of the sensitivity between the fingertip and palm base influences the prediction of shapes.
Therefore, we prepared asymmetrical shapes
by rendering only the upper part or the lower part of the primitive shapes.
This study comprised three sets.
The first set is for familiarizing the participants with the device.
The data measured in the first set were not used for the analysis.
Each set involved ten trials,
through which the participants predicted the shape rendered by PoCoPo.
In addition, the answers,
correspondence between the rendered haptic sensation and intended shape,
were not revealed after each set.
There were no duplications of shapes and no time limitations for each trial.
The order of the presented shapes was randomized and counterbalanced across the participants in each set.
The results of the shape prediction with 10 participants were summarized in a confusion matrix.
The numbers in the matrix represent the total number of predicted objects in the final two sets of the study
when each "true object" was rendered.
The maximum number is 20, and the minimum number is 0.
Overall,
the participants were generally successful in predicting the shapes rendered by PoCoPo
with an average success rate of 88.5%.
Three shapes, namely,
the rectangle,
circle,
and semicircle (upper),
had a prediction rate of 100%.
The shape with the lowest prediction rate was the sphere.
The participants sometimes tended to be confused the sphere with the circle.
In the current implementation,
because the number of rows was only three,
the participants seemed to have difficulty understanding the height differences between the shapes.
This result indicates that
increasing the number of rows would improve the prediction of shapes.
The prediction rate of the three upper shapes was 93.3% on average,
and that of the three lower shapes was 83.3% on average.
The participants tended to be confused the hemisphere (lower)
with the semicircle (lower)
and vice versa.
We assume that this result is due to the low sensitivity of the palm compared with the fingers.
The comments from the participants support this result.
If the users cannot notice the differences of shape with their palm base,
it might not need to increase the number of pins for the palm base.
Next, we present PoCoPo’s possible VR applications to present shapes and movements.
PoCoPo can render static objects by controlling each pin length.
These figures show how the user can hold several shapes in VR.
PoCoPo can represent a glass with a linear surface,
a matryoshka with a convex surface,
or a trophy with a concave surface in VR.
By dynamically moving each pin length in the user’s hand,
PoCoPo can make users feel as if they are holding moving objects.
In this example,
the user feels the pulse of the hamster,
which is represented by the expansion and contraction of the pins of PoCoPo.
In this case, the user can feel like a snake is wiggling in her hand.
Of course, there are several limitations and future work to be addressed.
Compared with those of other pin-based shape displays,
PoCoPo’s pin speed is slower,
the number of pins is smaller,
and the display area is limited.
A possible solution for these limitations is to utilize a visuo-haptic illusion.
Although such research has been conducted with a grounded pin-based shape display,
we believe that some of these techniques can be used in handheld pin-based shape displays as well.
Finally, PoCoPo lacks touch detection capability.
Some of the existing pin-based shape displays enable each pin to detect a user’s touch independently by using capacitive touch sensors.
By adding touch sensing capability of PoCoPo,
we believe that we can expand its design space significantly.
Ok, let’s move to the conclusions.
The main contributions of PoCoPo are threefold:
First, we proposed the concept of a handheld pin-based shape display
for providing skin contact sensation on the user’s hand in VR.
Moreover, we present the design and implementation of PoCoPo
that miniaturize the size of the pin-based shape display into a handheld size.
Finally, two user studies were performed to understand the capability of PoCoPo
through investigations of the prediction of the shape rendered by PoCoPo
and the visual size acceptance range.
If you have any questions and comments, feel free to email me.
That’s all for my presentation. Thanks.