Hi everyone, my name is Dishita Turakhia
and I am PhD student at MIT with Prof. Stefanie Mueller,
and I am very excited today to talk about Adapt2Learn-
a toolkit for configuring the learning algorithm
for adaptive physical tools for motor skill learning.
Let me begin with recapping some of the prior work
in the area of motor skill learning with adaptive physical tools.
We presented this work at TEI recently,
where we showed that adapting tools based on learner’s performance,
can significantly improve results in motor skill training.
So for example in the case of learning
to throw a basketball in the hoop,
we built an adaptive basketball stand
that adjusts its hoop diameter
and adjusts its hoop height
as the learner improves in their skill.
The goal is to use physical adaptation
to adjust the task difficulty such that
it is neither too hard, nor too easy for the learner.
And our studies showed such auto-adaptive training
led to higher learning gains when compared to
conventional static and manually adaptive training methods.
Our key contribution for achieving this auto-adaptation,
was the learning algorithm that we developed.
This algorithm sensed the learner’s performance,
and adapted the tool to maintain the task difficulty
at the optimal challenge point at all times.
So while in our prior work,
we showed that physical tools that adapt
using our learning algorithm
can help in motor skill training...
The next question that remained unexplored was –
how can we support designers in the design and creation
of such adaptive tools for various motor skill learning.
We address this question in our current work where…
we first conducted a formative study
to understand the challenges designers face
in creating adaptive training tools.
We then built the Adapt2Learn toolkit
to support designers in addressing those challenges,
and finally, we built several applications
to evaluate the generalizability of Adapt2Learn.
Let us start first with the formative study
We conducted the study for 8-weeks
in a bi-weekly design studio format
with 32 participants grouped in teams of two.
We asked the teams to design, prototype, and build
an adaptive training tool for a motor skill of their choice.
By letting the teams choose the motor skill themselves
we were able to explore the design space of adaptive training tools
for various motor skills and identify
the common challenges in building them.
Participants started with brainstorming 10 ideas each
and then selected one idea per team that they used
for designing and prototyping their tool
and finally, for user testing.
During the 8-weeks study, we observed their
design process, prototype implementation, and the algorithm development.
Finally, we conducted semi-structured interviews
with 8 volunteering participants.
And we were quite impressed with the
variety of creative designs developed for
a wide range of motor skills.
The applications ranged from sports,
for example, this adaptive skateboard that
sensed the learner’s balancing skills
and adapted the board length...
...to music,
for example, this adaptive piano
that sensed the learners note timings
and actuated the keys to lift them up or down.
There were applications in other motor skills
such as learning sign language using this adaptive glove,
and even walking in heels using
these adaptive heels, where
as the learner's balance improved,
the heel height is increased.
In the study, we observed that
the participants faced several challenges in different stages
of creating adaptive training tools.
In the design stage, the participants found
choosing appropriate sensors and actuators challenging.
Please refer to the paper for the design guidelines
we provide to help designers with this step
In the implementation stage, they faced the common challenge of
quick prototyping and building robust hardware.
We do not address this as there are several
toolkits in HCI to help designers with this aspect
Finally, we observed that the step that several teams found
most challenging was the step of creating the learning algorithm.
Only 5 out of 16 teams deployed an algorithm of some form.
Because this challenge is unique to motor skill learning,
our work focuses on supporting designers with this step
of creating the learning algorithm for their designs.
In particular, the teams faced two specific challenges:
First, was configuring the learning algorithm itself.
This involved mapping the sensor values to the actuator values,
such that the performance caused the adaptation of the tool’s task difficulty optimally.
We observed that there was no standard way
of developing the learning algorithm
and one of the teams said that the biggest challenge was
optimizing the learning algorithm, which was quite brute force."
The second challenge was assessing if the algorithm used
was working optimally as desired.
As another team pointed out that,
"The problem was not knowing if they had come up with [an] optimal solution."
To address these two challenges,
we built the Adapt2Learn toolkit.
Adapt2Learn toolkit has two components:
First, to help designers with configuring the learning algorithm,
we built a user interface, that designers can use to configure
their sensors, actuators, and respective threshold values,
and our UI auto-generates the learning algorithm for their tool.
And secondly, to help designers with
evaluating the generated learning algorithm,
we built a visualization tool that
shows them how the tool is adapting according to
the learner’s performance in real-time.
I will now quickly demonstrate how the user interface works
with a walkthrough of the configuration process.
And I will use the example of adaptive basketball from earlier.
The first step for designers is to register the sensors
and the sensor states in the UI.
So, for example, we register the piezo sensor to detect board hits.
And register the piezo values for board miss as a "failure" state
and values for board hit as a "success" state
Designers then register additional sensors
i.e. a switch sensor to detect a basket hit.
And then register the switch value for basket miss as a "failure" state
and basket hit as a "success" state
The second step for designers is to register the actuators
and the actuator states in the UI.
So for example, we register a servo motor to tighten or widen the hoop.
And register the servo rotation values for
hoop widen as a "failure" state and
hoop tighten as a "success" state.
Similarly, designers register additional actuators
i.e. a stepper motor to raise or lower the stand.
And register the stepper motor rotation values
for stand lower as a "failure" state and
stand raise as a "success" state
Finally, designers define the evaluation units and
generate the code
This code is a microcontroller script with our learning algorithm
that is auto-generated and customized for the physical tool.
It can now be deployed to a microcontroller
and used to adapt the tool for motor skill training.
To evaluate if this auto-generated code with the learning algorithm
is adapting optimally, we also built a visualization tool…
that plots in realtime the chart of the learner’s performance,
their improvement in scores, and the moments when
the tool adapts to a harder or an easier setting.
This visualization helps designers in assessing
the timing of adaptation with respect to the learner’s skill levels
and check if the tool adapts optimally.
To test if Adapt2Learn generalizes for various designs
and sensor-actuator combinations,
we built several application examples.
We built and configured an adaptive armband
with a single sensor-actuator combination.
This armband supports learners in keeping
their elbow straight during a golf swing.
It has a flex sensor to detect if the learner's elbow is straight or bent,
and a pneumatic pump that deflates and inflates the arm band
to restrict bending of the elbow.
We built and configured an adaptive wobbleboard with
a combination of two synchronized sensors and one actuator.
The wobbleboard has an inflatable cushion that
supports learners in learning to balance the board.
We mounted two ultrasonic sensors on opposite sides of
the board to detect if it is stable or wobbling,
and a pneumatic pump to deflate and inflate
the support cushion that restricts wobbling.
We also built and configured a bike with a combination of
two synchronized sensors and
two synchronized actuators.
The bike has adaptive training wheels that help
learning to balance the bike.
We mounted hall-effect sensor on each of the training wheels
to detect how often the training wheel is used,
and we mounted stepper motor on the training wheels
to lower or raise the wheels based on learner’s performance
Please refer to the paper for more details on the implementation
and additional examples from the user study.
In the future, we would like to expand the types
of sensors and actuators that are supported in our UI.
which is currently, eight types of sensors
and four types of actuators.
We would also like to add the functionality to be able
to re-configure the algorithm in real-time during the training.
And finally,
we plan to evaluate the usability of the Adapt2Learn toolkit
with designers during different phases of the design.
In summary, in this work, we highlighted that the key challenge
in the design of adaptive training tools
is deploying the learning algorithm.
We showed how Adapt2Learn can support designers in
configuring the learning algorithm for their respective designs,
and how Adapt2Learn generalizes for various designs
and sensor-actuator combinations.
With this work, we can support designers to
further explore the design space of adaptive training tools.
With this, I look for to the discussion
and am happy to take questions.