Welcome to my presentation on
Optimal Control for Electromagnetic
Haptic Guidance Systems.
This work was done with colleagues at
ETH Zurich
and New York University.
We propose a novel time-independent
control strategy for electromagnetic
haptic systems.
Pen-based interactive systems, that
feature haptic guidance,
support users in a variety of
applications,
such as drawing, sketching or CAD design.
The goal of such systems is to enable users
to draw
higher complexity shapes with less effort
and higher accuracy, or to support users
through virtual haptic tools such as
rulers or guides.
Crucially such systems aim to strike a
balance
between giving users a strong sense of
control and
agency.
Here we see a prototypical version of
our system and
how it can be used and the different
components of the system;
such as a pressure based tablet and an
electromagnet
mounted on a biaxial linear stage.
Yamaoka and colleagues proposed dePENd
here at UIST in 2013.
Their system uses a permanent magnet on
a biaxial linear stage.
However they use an open-loop controller.
This means that the user can only react
to the system and not vice versa.
Due to this the user needs to passively
follow the actuator.
Korres and colleagues proposed KATIB
which is a haptic-visual guidance system
they use, unlike dePENd, a closed-loop
control strategy.
However, this is a time-dependent
strategy.
This makes adapting to the drawing speed
of a specific user
for instance impossible. Thereby still
limiting
user autonomy.
Basically both system consist out of a
desired path,
and furthermore there is the current pen
position.
On the path there is a setpoint or
basically the point where the user
should be;
the desired point.
The goal is to provide haptic feedback
from the pen to the desired point.
The easiest way to do this is to have
the electromagnet
on top of this line.
However, due to the sharp decline in
electromagnetic force,
the electromagnet needs to be
sufficiently close to the pen.
Let's discuss the difference in
behavior of the system between open-loop
and time-dependent closed-loop control
to really understand
the problem at stake. On the left we see
an open loop controller.
With an open loop controller we can see
how the setpoint and electromagnet
move independently from the user
position.
Likely being even so far away that a
magnetic force
cannot be perceived by the user anymore.
Rendering the system only useful if the
user is fully compliant.
On the right we see time-dependent
closed-loop strategy.
Although the desired setpoint behaves in
a similar fashion as in the open loop
controller
the electromagnet stays within actuation
range of the pen.
However, since the setpoint moves this can
lead to haptic
feedback in the wrong direction; such as
in t=3.
Here from a user perspective the goal
is to have assistance during pen-based
tasks.
This feedback should be natural and the
control should remain with the user.
From this goal we can deduce two
important factors.
Natural would indicate that the tool
needs to be untethered
as well as minimally instrumented as
possible.
Electromagnetism is a logical solution
for untethered tools.
With control the system should adapt to
the user and not the other way
around. This gives rise to the question
on how we can control haptic feedback
systems especially electromagnet based
to find the balance between haptic
feedback and user autonomy.
To this end we propose a real-time
closed-loop control approach that allows
users to retain
agency and control while being assisted
by an electromagnetic haptic guidance
system.
We contribute a novel optimization
scheme
that enables formalizing this problem.
On the right we visualize how with our
control strategy
the setpoint and electromagnet are
dependent on the user.
In the slow motion shot we show why open
loop controllers provide problems,
especially in magnet based systems. Due
to the sharp decline in magnetic force
over distance
extreme snapping behavior can occur. We
can see that this becomes fluent
with our method.
In our paper we make two distinct
contributions.
Our first contribution is the novel
control strategy.
The second contribution is a
prototypical hardware implementation
which allows for a variety of
applications and user evaluation.
Let's first discuss our control strategy
in more detail.
In essence the system consists of an
electromagnet,
which has both the position and
intensity,
as well as a desired path. Furthermore,
there is the current pen position
of the user. The goal of the strategy is
twofold.
First of all, to minimize the distance to
the desired path,
and, secondly, the actual force should be
the desired force.
Meaning the correct intensity in the
correct direction.
This is a hard problem. Due to the
snapping
our method needs to be fast and precise.
Our control strategy is based on model
predictive
contour control. tthis is an iterative
method with the receding horizon,
is time independent as well, and is often
employed in quadcopter control or CNC
machines.
Basically, in an MPCC we minimize a cost
term.
this minimization is
constrained by a systems model.
We're trying to find the best inputs and
states of the systems,
these are U and X relatively. For an
input you can think of
acceleration where state is position for
instance.
In our case we have three cost terms. The
first cost term
finds the closest point on the path with
regard to the user current position.
The second custom promotes overall
drawing progress,
and the last cost term covers the
position and intensity of the
electromagnet,
which results in a force onto the tool.
Let's first of course discuss the cost
that governs the path following.
For the path following error we do not
have to take into account the
electromagnet.
This leaves us with just the path and
the pen position.
The goal is that we find a point on the
path that minimizes the euclidean
distance between
the path and the pen. However this is an
optimization problem on its own
and time-wiise infeasible to do in an
interaactive system.
and frankly since the gradient is not
always present
this is impossible.
Therefore we introduce two new lines.
The first line the dotted
is a tangent at the point on the spline.
The second one is orthogonal to it and
intersects
with the pen position.
Given this we try to minimize the lag
error shown in red.
This is the projection of the pen onto
the tangent.
The second error is the contour error
denoted in blue
which is the line orthogonal to the
tangent.
If the lag error approaches zero the
contour error
approaches the euclidean error.
Then we simply sum the two to get the
path error.
The path following cost gives us an
approximation of the closest point on
the path.
However natural correct will not say go
towards the closest point,
but slightly forward. This is where the
progress term comes into play.
The desired point can also be defined in
terms of theta,
or the relative position along to the
path, here
a higher theta means that it is further
along it.
Therefore we can generally reward a
higher theta.
Finally, there is the force cost term.
In our case the magnet is always
attracting the pen.
This happens from a center of a
permanent magnet embedded in the pen
in a straight line to the center of the
electromagnet.
For sake of simplicity we
will forget
about the z-component of this force.
This force is mainly depending on two
factors. The distance between the two
magnets
as well as the intensity of the
electromagnet.
The intensity can be controlled by the
current that is used to power the
electromagnet.
This describes the actual force.
The second element of our cost is the
desired force.
This force is in the direction of our
desired point on the path.
The final cost term then is the squared
difference between the desired
and the actual force. The system will
change the position
and intensity of the magnet in such a
way that this is minimized.
Calculating the force between two
magnets is not straightforward.
However, crucially this enables real-time
control.
For those who interested we derive an
analytical equation to do so in our
paper.
Now that we discussed our proposed
method let's take a look at the hardware.
We implemented on the system consists of
an electromagnet mounted on a
bi-axial linear stage.
The choice of both is not trivial we use
FEM analysis to inform our magnet
choice.
This linear stage needs to be needs to
provide enough force for fast
accelerations.
The system is agnostic to the tablet
that is used to determine the pen
location.
This can be pressure based thereby
allowing for pen and paper,
Or tablet which allows for visual
feedback.
Furthermore the stylus and pen used need
to be augmented
with a permanent ring magnet.
With our proposed method and
prototypical hardware in place let's
test it on users.
We did two experiments here. First of all,
we tested our system with end users
on general accuracy. Secondly,
we used a set of different users to
compare control strategies.
For the comparison with different
strategies please refer to the paper.
Let's discuss the accuracy of the end
user.
We asked 12 participants to draw six
different shapes,
with and without haptic feedback. The
users received minimal visual
support during this task,
here denoted in red. The order
was determined by a latin-square so
that each participant drew half of the
shapes first
with haptic feedback and other half
without.
The order of the shapes was also
randomized.
Here you see a selection of results; one
per shape.
The orange is with haptic feedback and
the blue with out.
We did not apply any filter on these
and these are raw points the video is
sped up
at the factor of four.
Notice how the users with the feedback
are staying closer to the target
trajectory.
When we take into a take a look at the
data a straight line is the only result
that is insignificant.
The distance over here is the distance
to the closest point on the path.
In all cases, but the line, we can see how
the tails of of the without feedback
cases are significantly longer
it's an increase in maximum error.
Overall our systems make users up to 2
times more accurate.
For more details please refer to the
paper.
So what we can learn from our user test
is that more complex shapes benefit most
from feedback. Furthermore we noticed that
users,
due to remarks they made during the
experiment, are still limited by the
speed
of our system, and ,finally overall people
improve their accuracy.
Now that we have taken a look at the
evaluation let's discuss potential
applications.
As shown here one of the applications
could be to help users with calligraphy.
A second application could be to help
people with
outlining inking of complicated sketches.
In this case a dragon is drawn by an
novice user with no drawing experience.
Notice how the system can deal with
different line segments
and the lifting of the tool as well as
the swapping
of the tool.
These illustrations are final results
drawn by the same user.
The system here has the same settings.
This illustrates how the user is still
guided to the back to the same
primitive shapes.
Yet, they remain to have creative freedom.
Finally we show a use case with visual
support.
Here our users manipulating virtual
tools in order to support them with the
design of a chair.
Let's discuss our findings and head
towards the conclusion.
In interviews some users indicated that
they had the feeling that their drawings
without feedback
were more accurate once they had
experienced haptic guidance.
This hints at the possibility of
short-term muscle memory
when using our haptic guidance approach. In
terms of hardware the speed of the
employed stage was a limiting factor,
without which users would have been able
to complete runs and general inputs
faster.
In this paper we have proposed a novel
optimization scheme
for electromagnetic haptic guidance
systems based on the mpcc framework.
Our approach balances user input and
system control so that users can adjust
the trajectory and speed on demand.
We implemented our method on
prototypical hardware
and finally evaluated the system with
users.
All blueprints and code are available on
our website.
Thank you.