I’d like to present our work entitled Dynamic motor skill synthesis with human-machine mutual actuation.
I’m Azumi Maekawa, from the University of Tokyo.
Humans have extraordinary abilities for physical motions and dynamic movements. Athletes perform sophisticated motions, utilizing even slight information from sensory organs.
Thanks to recent improvements in computing powers and actuators, the performance of robotic systems has come to match or in some cases exceed the capabilities of humans.
However, when a robot needs higher power, its size and weight must be bigger. Therefore, it’s almost impossible to keep its mobility
So how can we integrate robotic capabilities with us as an extended body?
We’d like to start with Human actuation concept: it’s using people as substitutions to motors and mechanical components, as proposed in previous research.
In the digital fabrication field, similar concept also have been proposed. This technique allows the user to utilize computational process advantage and the systems utilizes human’s power and mobility.
Our basic idea is applying these techniques to human augmentation field.
Inspired by these works, we propose an approach called Human-Machine Mutual Actuation, where robots act with human actuation synchronously.
In this approach, robots and human exploits their each capability.
Based on this proposed approach, in this study, we aims to investigate the feasibility of the proposed approach and reveal technical issues.
As a first step, we focus on throwing motions, particularly throwing underhand, because it is a basic physical skill and requires dynamic movement with muscle force.
We validate the feasibility and explore important factors for further research through prototyping and user studies.
Next, I’ll explain the design of our developed system.
How can we control trajectories of a projectile?
The projectile trajectory is defined by the laws of physics according to the initial position and velocity at the release point.
So our basic idea is based on the assumption that we can get the desired trajectory by controlling only the release timing.
There are some approaches to implement the system to control the release timing of human.
One is using a handheld-type device and controlling the device motion. In this approach, users contact with an object via a robotic hand.
Others are exoskeleton, electrical muscle stimulation approach, and so on.
In this study, we selected a handheld device approach for its little burden on the user, and it will make the system test easy.
Our system consists of the handheld device, and a control software that tracks the device’s motion in real time. The handheld device has a gripper for holding and releasing a projectile with computational triggers.
The device is tracked by external motion capture cameras and an onboard IMU sensor. While the user is performing the throwing motion, the computation of the projectile trajectory is running based on the device motion in real time.
At the timing when the computed trajectory passes through a defined target area, the gripper is opened, and the projectile is released. In other words, the object is not released while doing a motion that misses the target area.
In this manner, the system allows the user to throw a projectile toward the defined target.
This is our developed handheld device. This employs one actuated DoF and acts as a release controller with an external trigger.
A button interface is installed to control the gripper’s opening and closing manually.
Using this device, we can control the user’s throwing timing externally.
In auto mode, the release is executed automatically according to the user’s motion.
To release an object at the optimal timing, we must detect the position of the device in real time.
Since human throwing motion is a dynamic movement, the precision of the projectile trajectory can be affected by the difference of the release timing in a millisecond order. So, we developed a real-time position tracking workflow.
In a preliminary study, we identified that the position error caused by the latency significantly degrades the system performance. We measured the end-to-end latency, and we found it was about 40 to 50 milliseconds.
We consider the latency consists of mainly two parts: the communication process between the computer and microcontroller, and process that accelerate motor to released the object.
We estimate the motion in the former process by the integration of the value of the onboard acceleration sensor.
About the latter, we decided to predict the future motion based on the assumption that the user’s arm movement can be regarded as constant acceleration motion in a short time.
In this way, we implemented the prediction process to suppress the effect of the latency. We adjusted and determined the ratio of these two factors empirically.
Next is user study.
We conducted user studies to validate the feasibility of the proposed approach and to understand the users’ subjective experiences. We evaluated the system performance by measuring the landing position of the projectile and collecting the users’ feedback.
First, in the situation where the user was aiming at the target, we compared the case where the user manually controlled the release timing and the case where the system controlled the release. We also analyzed the qualitative aspects based on the user’s feedback in this study.
In the second study, system performance was investigated in a situation where the user performed throwing without aiming at the target.
To observe the user’s behaviors, we demonstrated the system at a public exhibition, and analyzed the users’ feedback and experience. Please refer to our paper about the detail setup and procedures.
First, we describe user study1. The goal of this experiment is to evaluate the performance of the developed system in a situation where the user aims at the target.
We also investigated the user experience by analyzing the subjective reports. The diagram shows the experimental setup. It includes three targets on the floor and a marker as a guide.
In this user study, the system had two modes: one was auto mode and the other one was manual mode.
In the auto mode, the projectile was released automatically with the computational control according to the user’s motion. In the manual mode, the projectile was released by the user’s button push.
We aimed to investigate not only the performance but also the effect on the user experiment caused by adjusting the release timing with the system. So we asked the user to push the button in both modes and investigated how much the user could recognize the intervention of the computational process.
This is the result. We collected four-hundred eighty-six trials in each auto mode and manual mode.
We found that the system performs throwing selectively according to the assigned target. We observed the projectile tends to land on the near side of the target positions.
We concluded that one of the reasons for this can be errors of the model adopted in our system and hyperparameters in the prediction process.
We also found that the variances of the results for each participant tended to be smaller than that from the total trials of all participants. We concluded that this can be because the system was not optimized for each user.
We describe the qualitative results briefly based on the interview and the user’s feedback.
We found that the users were able to judge the mode from the results of throwing. On the other hand, the users couldn’t judge from the release timing difference.
We found that the system can create a situation where the operation is performed by a computer process even though the users think that they are performing by themselves.
Next is User study 2. The goal of this experiment is to evaluate the system performance in a situation where the user does not aim at the target.
The experimental setup was same as User study1. In this experiment, the system was executed in the auto mode in all trials.
To ensure that the projectile gets sufficient kinetic energy, the participants were instructed to swing their arm aiming toward a marker.
The participants were not informed of the target position of the system aiming for each trial.
Compared with User study 1 result, we found that the group without aiming at the target could throw more precisely. This is probably because the constant acceleration model used in our system is better suited to swing aiming toward the far side of the target.
In the current system, the computed projectile trajectory and the actual one do not match in the order of a centimeter accuracy.
This can be caused by various factors such as measurement error, unexpected user movement, variations of the contact condition between the projectile and the gripper, and so on.
For an informal validation study, we demonstrated the developed system at a public exhibition to observe the users’ behavior and obtain feedback. The right-side figure shows the demo setup.
Through this demonstration, we got the users’ feedback and lessons learned. We skip the details due to lack of time, but please refer to our paper.
Throughout the results of the user studies, we revealed several challenges to be tackled for designing systems based on our proposed approach.
First, we must consider how the system suppresses disturbances on computational processing caused by unintentional human motions.
For example, when the user performed an arm-twisting motion to throw, we found that the gripper affected the behavior of the object being thrown out. For dealing with such human disturbance, active stabilization techniques will be effective.
Second, the system needs to particularly construct a model of each user’s individual motions with an appropriate latency setting.
Through developing the system, we identified that humans can accurately perform their dynamic motor skills with their self-motion prediction models.
Therefore, a user-specific calibration will be highly required, and utilizing advanced motion prediction models or machine learning-based techniques will be promising way.
Third, we need to develop a method of inducing humans to move “correctly” as defined by the system.
Our observations suggest that a user’s undesired behaviors might lead to a significant decrease in system performance. We consider that one way to address this issue can be to design an incentive that occurs in the user experience.
Our results indicate that the user’s performance can be controlled without losing his/her sense of control. This suggests that a appropriate designed system can provide a sense of achievement or motivation to the users and induce them into doing desired behaviors defined by the system.
Finally, this is a summary of this study. We proposed Human-Machine Mutual Actuation concept and investigated the feasibility of our concept with a case study.
We developed a system that allows the user to perform throwing with a robotic handheld device.
We conducted user studies and revealed several challenges to be tackled in further research.