Hello, my name is Ken Nakagaki, I'm a 5th year PhD student at the MIT Media Lab, Tangible Media Group.
This place is my home lab, and in this video I'm very very excited to share my latest research paper named HERMITS…
…which was explored and developed in my home lab during the lockdown 2020.
In this paper, we present an idea to enrich the interactivity of self-propelled robotic tangible user interfaces, using mechanical shell add ons…
...which are basically external passive attachments to the robotic tangible user interfaces.
In the field of HCI, research in actuated and shape changing tangible user interfaces have been widely explored to embody dynamic digital information through physical shapes.
Researchers have been investigating to develop a generalizable configuration of actuated TUI hardware that can provide a variety of functionality and versatility, with a single hardware.
On the other hand, power of motion and actuation in kinetic art and mechanical automata has been a great inspiration for us.
When you look into these works, each specific art piece or sculpture can only create very specific motions, but each of they can provide a very rich expressiveness in their motions.
So having these as backgrounds, we have this research question of how we can empower the actuated TUIs, in terms of…
1. richness of display, animation, and expression capability, as you seen in the mechanical automata,
and 2. adding the reconfigurability for a versatile tangible interactions, so it can adapt to a variety of applications.
To move step forwards, we've got the inspiration from biology — hermit crabs.
Hermit crabs, as you know, shelters themselves into empty seashells and interchange the shell, according to their body sizes as they grow.
Our paper, HERMITS, named after hermit crabs, intends to contribute to this effort by introducing an addition of passive mechanical attachments, defined as 'mechanical shells'.
While the self-propelled TUIs have limited interactivity, the passive attachments greatly extend the capability of the core hardware.
In the past, researchers have developed different types of tabletop actuated tangible user interfaces...
…where the tangible objects can move around on top of the desktop surfaces to embody digital informations.
Recently, there are a lot of research in an approach to embed the locomotion capability, or actuated wheels in tangible objects, including Swarm User Interfaces.
We define these approaches as self propelled TUIs in our paper, and seek for a way to extend their interactivity with passive material or modules.
Extending the interaction of generic interactive devices using passive objects have been explored in the HCI to extend the interactivity of smartphones, displays, or even 3D printers..
In the context of actuated TUIs, activating passive material have been widely explored using shape displays as well as self-propelled TUIs.
For example, researchers have explored ways to simply handle solid objects moving them around.
There are investigations in assembling blocks into different shapes.
Animated craft is another one to create playful storytelling applications.
In our work, we focus on extending the motion-based interaction with passive module embedded with mechanisms, that has been partially explored using shape displays.
There is an open research space to explore this idea with self-propelled TUIs.
As a quick summary of the contribution of HERMITS — we introduce a general methodology to enrich the interactivity of self-propelled TUIs with mechanical shell add-ons.
We introduced the definition and design space of the mechanical shells,
…and we talk about the implementation of the proof of concept system prototype.
We present three application spaces to demonstrate the concept.
And lastly we show an open research space to motivate this general approach for the future.
Mechanical shell is a term we defined in our paper as “external passive attachments to self propelled TUIs that can be automatically attached, or detached.”
As you can see, mechanical shell can have a variety of form factor or interactivity for the self-propelled TUIs to dynamically or selectively attach and detach to the other.
While each mechanical shell can provide a very specific interactivity and functionality...
...as a whole, the system architecture can provide a very versatile, adaptive and reconfigurable interactivity to the users by interchanging the mechanical shells.
The overall design of hermits would look like this.
The system includes the self propelled TUIs controlled by the computer.
And these robots can be actuated to dock to the mechanical shells, to create the reconfigurable interactivity to the users.
To explore a variety of mechanical shells design, we have defined a primitive design spaces which became the basic vocabulary for design of the mechanical shells.
For example, 'shape' is the simplest property to be converted by interchanging the mechanical shells.
They can also extend light output, by using optical fibers, to extend the indicator LED of the hardware.
Extension of the motion is the key property in the design space of HERMITS.
For example, the rotation of the robotic hardware can be converted to vertical linear motion.
Also, the speed and torque of rotation can be converted to achieve specific tasks such as generating wind or walking over uneven surfaces.
Mechanical shells can also extend the sensing capability of robots, for detecting tangible input, which can be utilized for dynamic physical affordances,
— in this case, we are showing knob and slider examples.
One of the very unique feature of HERMITS is that multiple module of self-propelled devices can cooperate together to activate single mechanical shell.
With this capability, for example, the robotic devices can achieve rather complex interactivity or motion with multi degree-of-freedom, for robotic application or I/O devices.
Additionally, the multiple robots can work together to assemble, or to generate greater speed or force either by stacking to each other and rotate or by pushing single mechanical shell in parallel.
This design space became the basis for a variety of mechanical shell examples we created and demonstrated in our paper.
As for the implementation of the system, we employed and modified an off-the-shelf two-wheeled robotic toy named 'toio' to prototype the concept of HERMITS.
One of the primary modifications, we made is the addition of vertically moving pin for docking.
When they ride into the mechanical shells, they can have a robust connection to them, embedded with female docking slots.
With this, even when a user pick up the device, the robotic device won't fall off from the mechanical shell.
Using the toio API, we have developed the control architecture.
Single Raspberry Pi can control up to five toio via Bluetooth but we can use multiple Raspberry Pi to be controlled by a single computer to handle the input and output.
It's basically a scalable system that allows us to create a very large scale, interactive setup to explore a variety of interactivity.
Also, one uniqueness of toio is that they can detect the absolute position on the special mat by the camera embedded in the toio.
On the other hand, we also developed peripheral functionalities.
For example, the system can track the position of mechanical shells, using a camera and markers.
We have also employed joysticks to manually control toio robots for prototyping different interactivity which we used when exploring applications.
We have also defined 4 primitive designs for mechanical shells, which became the basis for our prototype to convert motions in different ways.
Each of the four design provides a different characteristics to convert the interactivity or motion of the robotic TUIs.
The details are summarized in our paper, but these became the basis for us to develop a variety of mechanical shell examples.
In terms of the applications, we have developed three different application or use cases in HERMITS.
Firstly, HERMITS can be used for physical space reconfiguration, for example for an adaptive and assistive desktop.
We believe this is very promising directions to be deployed beyond the desktop to other physical space in our everyday life...
...because people are already having these robotic vacuum cleaners, for example, that are moving around our space.
It’s really exciting to imagine how we can fully expand the capability and functionality of such robotic system in our physical environment by adding passive mechanical shell attachments.
It can also be used for interaction with digital data.
In our paper, we discuss the use for tangible mobility simulation where mechanical shells can be interchanged to...
...represent different vehicles or agents in simulated urban model but in dynamic tangible form.
Usually these simulations are shown or interacted only in screen, and we are excited to think of how this can be physically and dynamically embodied with HERMITS.
Animated storytelling demonstrates to convey story by interchanging the mechanical shells where each robotic module acts as an actor on the stage.
They can switch the mechanical shells just like changing costumes in the back stage in theaters, and appear on the front stage for interactivity.
As for conclusion, in our paper, we proposes HERMITS and defined the concept of mechanical shell.
With this concept we have developed both the overall system design as well as the design space of the mechanical shell.
Based on this design space, we have demonstrated a variety of mechanical shell examples.
We prototyped and introduced the implementation of our proof of concept system, and we described application spaces to motivate this new concept.
So with this, we are very excited that this idea is a first step towards a much bigger open research space.
For example, the future research space could include updating the self propelled devices to other hardware...
...such as drones which can freely move around in the 3D spaces to dock to mechanical shells.
What if the mechanical shells can be fabricated on demand, instead of only having a pre-fabricated and assembled mechanical shells.
We also need advanced control algorithms to orchestrate the swarm behavior to be able to dock and undock as an architecture.
We are very curious to look into doing user study and explore the higher level interactivity with this new hardware configuration.
Lastly, this research has been explored under the vision of Radical Atoms which envision the future of physical material that is coupled with dynamic digital information.
We believe the idea of HERMITS is imposing a very interesting perspective or paradigm in the Radical Atoms research.
In the past, there have been investigation in two primary approaches to develop this idea of digitally empowered dynamic physical materials.
One is self actuated machines embedded with motor, power and computation.
And another is passively activated machines that is activated in response to external stimuli, but with no embedded motor or computers.
I believe our paper HERMITS, in high level context, brings a new perspective in novel hybrid architecture, coupling these two approaches to be reconfigurable for adaptive interaction.
We believe there's a lots of research space to be explored along this general approach.
With that, I would like to thank my co-authors, including Joanne Leong, Jordan Tappa, João Wilbert, and Professor Hiroshi Ishii.
And if you have any comments or questions, please feel free to reach out to me.
Thank you very much.