Hi and welcome to my presentation on Omni.
We present a method for
volumetric sensing and activation of
passive magnets,
especially for haptic feedback systems.
This work was done in collaboration with
Juan and other colleagues at ETH Zurich
virtual and augmented reality are
getting more and more momentum
this is mainly due to
the promise for immersive experiences.
Current systems are mostly focused on
rich visuals to cater towards the end user.
However to increase the immersion into a
carefully crafted
artificial world it is necessary to also
support the sense of touch,
and to provide tactile feedback when
interacting with virtual objects
in 3D.
A lot of prior work exists on this.
Ranging from mechanical contraptions to
vibrotactile stimuli,
also larger electromagnetic surfaces
have been explored.
Especially relevant is NormalTouch by
Benko colleagues.
They propose actuated handheld
controllers,
that render the shape of virtual objects
through physical shape
displacement thereby enabling users to
feel
3D textures, surfaces and forces
that match the visual rendering.
Also relevant is Butterfly haptics of
Berkelman and colleagues.
The large problem with both of these
systems is that they require external
tracking
or are cumbersome.
In the end the goal of the user is to
have an affordable system
to deliver significant untethered forces
in a closed-loop fashion.
A keyword here is affordable. One way to
achieve this
for instance is to make the system
self-contained.
Secondly, untethered air-gapped
and free of mechanical contraptions.
A common solution for this is the use of
electromagnetism.
Finally the system should act on the
user.
This means that it should be able to
sense inputs.
All this gives rise to the question on
how we can leverage
electromagnetism for activation as well
as sensing.
To this end we have three contributions
in our paper.
A hardware contribution, we suggest a
method
of position reconstruction, and we
activate
at the same time. Let's first discuss the
hardware.
In terms of actuator there is some
related work. First of all, there is the
work of Patruska.
They propose an electromagnet with three
orthogonal coils.
Their square design, however, does not
create a homogeneous magnetic field.
Providing different perceived forces at
the same distance.
Secondly, there is the work of zarate
and colleagues.
They have a spherical electromagnet,
however the coils are not intertwined.
This results in less force
and more heating compared to our design.
Furthermore,
they require external tracking
In our system a permanent magnet is
embedded inside the tool.
The actuator a spherical electromagnet
and consists out of an iron core and three
intertwined copper coils.
Below that there is a PCB. On the PCB
are eight 3D hall sensors that enable
tracking of the tool.
Finally, the whole system is cooled by a
fan.
Now that we have an overview of the
hardware, let's dive right into the
details on tracking.
Tracking permanent and electromagnets
has been a popular topic.
Taylor and colleagues tracking multiple
magnets with analytical calculated
gradient. Chen and colleagues
can track multiple fingers and identify
them by attaching electromagnets to them.
Our approach differs in the sense that
we have an actuator in the same modality,
electromagnetism, thereby adding extreme
noise
to the tracking signal.
To understand our method let's start
from the beginning.
The first step is to measure a hall
sensor. This provides you with a magnetic
field at a certain location.
In this case it mainly consists out of
noise
made by the earth's magnetic field.
This magnetic field is influenced by a
permanent magnet
in a tool. However, it is possible to
approximate its influence
with the theoretical model, the so-called
dipole-dipole model.
This is denoted in blue and that is step
two.
Step three is to use a similar model on
the spherical electromagnet.
So now our hall sensor measures the tool,
the actuator,
and the noise.
Magnetic fields form a superposition and
therefore can simply be summed.
So the real-world measurements are
approximated by
the sum of our models as well as the
noise.
For the exact models please refer to the
paper.
We can move over the measured field to
the right hand side of this equation.
If our models are perfect this cost
should approach
zero. You can change our model
by moving the permanent magnet around
position p.
So our tool is likely located where this
cost is minimum.
We can do this by taking the gradient of
the cost with regard to the position.
Then we do simple gradient descent till
our cost and gradient approach zero.
For a more detailed algorithm please
refer to the paper.
However we specifically use a quasi-
newton method which is strong for
non-linear problems.
Here you can see the tracking as
observed through a mixed reality headset.
Since the method uses gradient descent
under real time optimization
it takes approximately 50 milliseconds
to converge.
We now have an actuator that can track a
tool but how do we actually provide
forces?
For this we can forget about the hall
sensors.
We just have a tool with a permanent
magnet and a spherical electromagnet.
The force in three dimensions can be
changed by altering the current.
The permanent magnet in the tool is
assumed to always point
towards the sphere so that the same pole
of the magnet always points towards the
center.
Let's now take a closer look at the
forces themselves.
You cannot only apply attractive forces
by having
the different poles.
By rotating the magnetic field of the
spherical electromagnet you can also
repel the tool or we can even create
tangential forces,
where the user perceives a sideward
force.
The forces can can be combined so
that the user perceives
a combination of forces.
More details on how to calculate input
based on the desired forces are proposed
in our paper. During experiments we have
measured forces
up to 2 newtons for 30 seconds.
Lower forces can be sustained
indefinitely.
This is enough to lift an apple.
Furthermore we show that we can rotate
the magnetic field
here with ferromagnetic fluid, but also
with a permanent magnet. In this case one
that is also found
in our tool.
Let's take a look at the accuracy we
can track.
For this we use the 10 camera optitrack.
We measure the tracking with and without
activation.
Here we see a violin plot that plots the
distribution of
errors. We can see that our tracking
without actuation
is around the 5 millimeter euclidean
distance.
With actuating this moves
up to a bit less than seven millimeters.
However, in both cases the error per axis
stays well below five millimeters.
Now that we have an idea on how accurate
we can track,
let's see what applications our system
enables.
One of the applications we propose is
for designers.
Here a person is extruding from a shape.
While extruding the person perceives
attractive forces.
Another option is to explore non-rigid
bodies. Here
a person is bravely poking a dragon,
upon impact the user perceives a force
away from the dragon.
Finally our system can be used for
gaming.
Here in a joystick like fashion. Upon
impact on the wall
the user perceives a large attractive
force.
To conclude, there is a future work ahead
of us.
One of the things is that we need to
improve the tracking. This can be
done by improving
the model we have of our world.
Our method can also be extended to track
multiple tools.
This could potentially open up more use
cases.
And finally we need to evaluate the
whole system with users.
To summarize, in this paper we introduced
new hardware that is capable of tracking
and actuation. We introduced the method
for position reconstruction,
and showed how this can be used during
actuation.
All blueprints and codes are available
on our webpage.
Thank you.