Hello, my name is Changyo Han from the University of Tokyo.
This was supposed to be my first CHI participation so I feel so sorry that the event is canceled due to COVID-19.
I am staying home safe, and I hope everyone's safe.
This work, PneuModule is a part of my PhD project and it's done by my co-authors: Ryo Takahashi, Yuchi Yahagi and Takeshi Naemura.
First, I want to explain the motivation and the overview of this project.
It builds upon the vision presented by Nicolas Villar, that user interfaces have to be malleable, which means that their forms and layouts have to be adaptable to the demand from users.
Their prototype consists of a surface that allows physical controls can be dynamically placed, and they can be instantly recognized owing to the embedded microcontrollers and wired communication channels.
This kind of spatial rearrangement of physical controls can be found on tangible user interfaces.
Tangible interfaces inherently have features to recognize the position of the tangibles on the interactive surfaces.
We can interact with the system by spatially arranging the tangibles, but it is often difficult to implement physical input capabilities to the tangible objects.
Based on the aforementioned motivation, we propose PneuModule.
PneuModule is a modular physical interface platform that works on pressure-sensitive touch surfaces.
The layout of the physical controls can be instantly reconfigured owing to the inflatable pin arrays on the bottom of the modules, which can be actuated by passive pneumatic mechanisms.
Owing to this configuration, the system can support both rigid and deformable user inputs.
I will then explain the related work.
In the field of HCI, there have been a body of research that aims to make instantly reconfigurable physical interfaces.
It can be accomplished by using electronics, or magnetometers, or RFIDs.
These kind of physical controls do not require complex wiring, users can explore and adapt the interface layout.
Using passive tangibles on touch surfaces offer both the spatial reconfigurability and the interactivity.
Ohmic-Touch and Flexibles used 3D-printed conductive tangibles that accepts user inputs.
Geckos presented tangible interaction capabilities on pressure-sensitive touch surfaces.
On our previous work ForceStamps, we presented a physical interface prototyping tool using pressure-sensitive surfaces and 3D-printed tangibles.
On the view of deformable user inputs, there have been plenty of works that utilize pneumatic pressure to accept user inputs.
PneuModule extends interaction capabilities upon these work.
Next, I will talk about the implementation of PneuModule.
We came up with the following design requirements.
First, we need a mechanism to transmit user input to the touch surface.
Second, we need a modular design to recognize the interface layout.
Third, we have to track the spatial position of the tangibles on the touch surface.
So we came up with the structure shown on the left.
Our system consists of two kinds of modules: a main module which can be tracked on touch surfaces.
And extension modules which have physical input capabilities.
The extension modules can be connected to the main module with airtight connectors, and can forward user inputs to the inflatable pins through internal air pipes in the main module.
The actuated pins touch the pressure-sensitive touch surface, thus enabling the user input can be transferred to the system.
The figure on the right shows the implementation of the main module.
It is assembled from some rigid parts – main body and the bottom lid – and the inflation layer is sandwiched between them.
There are four magnets to prevent the module from floating off the surface due to inflated pins.
Let me show you how the pin inflates from the video on the left.
As the user input is applied, the internal pressure of the air chamber increases and actuates the pin on the bottom.
This kind of input forwarding can be implemented by other mechanisms such as magnetic or mechanical ways.
But we decided to implement it in a pneumatic manner, since pneumatics can transmit the force continuously, and can be implemented with less complexity.
Next, to recognize the layout of the modules, we arranged the inflatable pins in a 2D array.
We arranged 45 pins which can be divided into 5 regions, and each region represents the state of the inputs coming from the corresponding side.
The 9 pins in each region can be used either for representing the ID of the connected module, or forwarding the applied user input.
We can see from the pressure image that it can be easily breakout into different regions, and with a simple computer vision algorithm, the intensity of the blobs can be extracted.
As the position of the pins are predetermined, the system can recognized the connected modules from the pressure image.
To track the main module, we placed the permanent magnets in a rotational asymmetric way.
It enables the system to track the position and orientation of the module.
We have two different options for the pin actuation: One is actuation plug, and the other one is thru-hole plug.
When a extension module is connected to the main module, the actuation plug actuates the pin in the corresponding position.
The thru-plug connects the air chamber of the extension module to that of the main module, thus forwarding the user input.
Next we explored the material and geometry options for fabricating the inflatable layers.
We compared two silicones which have different stiffness properties:
One is the popular Ecoflex 00-30 with a Shore hardness 00-30, and the other is HTV-2000 with a Shore hardness 18A.
We made a prototype with Ecoflex and since the material was too soft, we encountered the following problems.
First, the pins over-inflated even with small increase in internal pressure, so interfered with adjacent pins, making the system difficult to distinguish them.
Second, when we slid the module on the surface, the pins caused positional misalignment to each other.
Therefore, we chose to use slightly stiffer silicone HTV-2000 to restrict the amount of inflation.
After that, we explored various options for the geometry of the pins.
There are mainly two factors that affect the inflation of the pins: thickness and diameter.
With the shown design parameters, we fabricated and measured the inflation of the pins in two ways:
First, we visually inspected the height of the pins and next, we measured the transmitted force using the touchpad.
We used commercially available pressure-sensitive touchpad–Sensel Morph.
Among the design parameters, we found that the pin design with 5 mm diameter and 0.4 mm thickness shows the most considerable inflatable characteristic so we determined to adopt the parameters.
Next we designed the geometries of connector sockets and internal air pipes.
The dimension of the sockets are determined by the plunger we used.
We decided the geometry of the internal air pipe by considering the shown equation (internal pressure when the plug is fully inserted to the socket).
When the socket is fully compressed, it resulted internal pressure of 3.9 to 5.3 atm, which is suitable for actuating the pins as shown in the former slide.
To fabricate the modules, we used a SLA printer for the rigid parts since it is difficult to obtain airtightness from commercial FDM printers.
The inflation layer is cast by silicone and glued to the rigid body.
Then the magnets and bottom lids are assembled.
For fabricating extension modules, we printed rigid bases for connecting them to the main module.
Syringe plungers are used as plugs for achieving the airtight connection.
For user input, the air compression mechanism can be variated to offer different input modalities.
For example, these rigid input modules have internal cylinders for air compression.
For the press module, it is achieved by a linear motion, for the rotate module, we translated the angular motion to the linear motion by using a screw thread.
We also implemented extension modules that accept deformable user inputs such as squeeze, twist, and bend.
To fabricate the deformable inputs, we cast silicone into 3D-printed water-soluble mold and dissolved the mold in the water to make the internal air chamber.
Then we connected the deformable part to the base with a zip tie for airtight connection.
We also fabricated a extension module that has no input functions, but can alter the ID representation of the connected input module through it.
It has the thru-hole plugs except for one plug. This plug actuates a ID pin to make change of the ID representation.
Let me show you the operation of the modules.
They are instantly recognized when connected, and we can see that user inputs are forwarded to the touch surface.
For a multidimensional input, we can assign multiple pins to distinguish them.
And with the stack module, we can change the ID representation of the connected module.
Finally, I would like to show the application examples of the proposed system.
We implemented a shooting game that the users can change weapons instantly by swapping the physical modules.
Each module corresponds to a specific weapon, and the user can customize the spaceship for different situations.
For example, for this boss alien, the user can equip a charge shot to give effective damage.
PneuModule also can be a tangible representation for virtual characters.
Users can design the interaction in-situ by swapping the input modules.
In the video, the user swapped the stretching arm to the bending arm.
Next we made an adaptive music controller that enbables tangible reconfiguration of the musical components.
The clean guitar tone turns into distorted sound by inserting the distortion module, and additional controls can be added to manipulate the sound, like this volume knob.
Finally, I will discuss the limitations and future work.
First one is scalability of the system.
As you can see, the size of the main module is pretty bulky compared to other modular blocks such as LEGO.
The scalability is limited by both the resolution of the touch surface and inflation characteristic of the pins.
To realize this system in a smaller scale, we need a way to fit the resolution into the scale we aim for.
Second, the surface friction increases with the number of actuated pins.
This is because the silicone has a quite of surface friction.
In the demonstration, we mitigated the friction by using a dry lubricant but we hope there’s a better solutions for this.
The final one is the fabrication process.
We used different fabrication methods for rigid and soft parts.
But with sophisticated multimaterial 3D printing, that could be done in a single pass to avoid the cumbersome manual assembly.
Okay, let me summarize my talk.
We presented a modular physical interface platform using pneumatically-actuated inflatable pin arrays and pressure-sensitive touch surfaces.
We derived a marker design for tracking and recognizing the configuration of the modules.
We explored various inflatable pin designs for force transmission.
And showed diverse application examples of the proposed system.
If you have any questions, feel free to contact me via e- mail or Twitter, thank you.