- Thank you very much.
So a spontaneous spatial coupling
is a hybrid technique of motion correlation
and cursor based interaction.
It enables the appropriation of any body part
or movement of a tangible object
to be used as a pointing device on an ad hoc basis.
The technique can automatically account
for the user's range of motion,
and the pointing phase only needs to persist
for as long the user requires.
So a step by step breakdown.
So in the first instance,
controls are displayed to the user
in the form of moving targets.
So here we have six targets,
and three rotate clockwise,
three rotate anti-clockwise,
all with different phases, so each movement is unique.
And in order to interact with the display,
the user needs to mimic the motion,
and they can mimic the motion with any body part
or tangible object.
The system looks at its inputs,
which is the user's movement,
and the outputs, which is the moving targets,
and if there's a correlation between the inputs
and the outputs,
a pointing phase is generated.
In this case the user selects the channel
using the horizontal movement.
But it's also important that they can decouple
from the control,
so in this case we use the vertical movement
to select the channel,
and the system reverts back to the motion matching phase.
So we leverage two different techniques,
each with their own advantages.
So motion matching is highly discoverable and intuitive.
It is great for simple binary tasks
such as playing or pausing, muting the volume,
and it's suitable for multiple users,
and can be performed with any body part or object.
In contrast, cursor based interaction
is a familiar paradigm,
that we use in our everyday lives,
and it's good for continuous interactions
and object manipulations.
It too can be used by multiple users
with any body part or object.
But when we combine these approaches
we can leverage further advantages.
So the pointing is only activated
upon the selection of a control,
and only needs to persist
for the duration of an interaction.
So this effectively shifts the Midas Touch problem
we have with cursor based interaction
to the motion matching phase,
where we can control and design specific motions
to alleviate this problem.
We can use different body parts or objects
to instantiate pointers,
and the scale and range of the user motion
in the motion matching phase
carries over to the cursor based interaction.
So problems like gorilla arm are alleviated
because the user can switch input modalities,
and ill mappings tend to be avoided
because we can define the range of the user's motion.
We can also tailor the pointing
to the context of the interaction.
So depending on what type of modality the user uses,
and also depending on what task
they are trying to complete.
So we introduced MatchPoint,
which is a webcam implementation of the technique,
which accepts any type of motion as input,
and is built upon the TraceMatch system,
which was introduced last year at UbiComp.
And this is how it works.
So in the first stage,
we use feature detection and optical flow
to track unique features in the scene,
which we match against the moving targets.
So here we have yellow and green features.
Green features are points which exhibit enough motion
to be considered a candidate for the matching process.
We then use a Pearson correlation,
and fit a circle using the RANSAC method
to detect whether any of the features
have matched the moving targets.
In this case, we have two features.
We can use the ideal user's motion,
derived from the RANSAC process,
to calculate the CD gain.
So for example head movements
will result in much smaller motions than hand movements.
But we have a problem.
In the next stage we want the cursor based interaction,
and we need to track the object or body part
which matched the motion.
So we retrospectively look at the dense optical flow
to see which pixels match the motion of the target.
This allows us to initialize the pointing
and the tracking task,
and we then use a modified version
of a median flow tracker
to track the object or body part
for the cursor interaction.
So, there's a number of techniques.
The conventional pointing paradigm
of having a single pointer
control all the elements on the interface
can be leveraged by decoupling,
allowing the user to decouple from the pointer
as and when they require,
and to change input modalities.
However it doesn't really leverage
the advantages of the technique.
Having multiple pointers with multiple functionality
means different pointers
can provide different functionality,
and the CD gain can be mapped to controls
rather than interfaces.
So in a TV example,
you could have a TV guide
using a two dimensional pointer,
you could have a channel selection
using a horizontal slider,
or a vertical slider for the volume.
We can do similar things with a video player,
such as controlling the playback,
selecting the progress of the video,
and once again using the volume.
So here's an example of using the in-built camera
of a surface
to rewind a recipe
using the hand mixer.
The user doesn't have to stop what they're doing,
put the object down.
They can interact with the system
and carry on their task.
The problem with the previous approach is
you can't have multiple viewers controlling the volume
or selecting a channel,
it gets a bit messy.
But you can have multiple users
interacting with the same control,
which for example, you could use
to contextualize discussions.
So in this example,
we have two users who are contextualizing a discussion
on a house plan from a distance.
Both acquiring the same control
to dictate the discussion.
We can also have parallel pointers,
so a single user can acquire multiple pointers,
and this can be to complete tasks more quickly,
to enable more complex interactions,
to manipulate one control based on the state of another,
and we can categorize into three categories.
So the first is unrelated.
So the control of one pointer
does not affect the other.
So for example in a smart home system,
one might change the heating,
while changing the volume of the radio.
These are unrelated,
for these you could do them at the same time.
You could loosely coupled.
So the pointers affect the same object,
but could be used individually.
So in this example we've got two pointers,
one controls the panning of an object,
and one controls the zoom.
And a user can use different input modalities
to control the pointers.
So the head controls the zoom,
and the hand controls the panning.
We can also have tightly coupled pointers,
in which the interaction results
from the relationship between two pointers.
So in the same object manipulation example,
we can have bi-manual input,
where acquisition of two pointers
allows the zooming, panning, and rotating of an object.
So far we've looked at temporary bindings,
but tangible objects present a unique use case,
because they can act as an intermediary
between the user and a control.
Objects have interesting properties,
such as being able to be left stationary
for prolonged periods,
and they can afford specific manipulations
depending on the type of object.
So in this example we have a radio player,
where objects are assigned to different controls
such as changing the song,
the playback and the volume.
Each object is in turn coupled
to one of the controls,
and this time,
a persistent binding is created,
so that manipulation of the object
results in manipulation of the interface itself.
So this opens up a new design space.
We've provided initial framing with a few examples,
but this supports a wide range of applications,
as we can appropriate anything the user can move.
So we can have combination of body parts, objects,
more persistent object bindings.
The users can manipulate multiple controls at once,
and you can one or multiple users doing this.
And this opens up for the exploration
of mappings for specific contexts
rather than general purpose mappings.
Input can be uniquely tailored to the context,
depending on the method of input the user decides,
the calibration of the CD gain,
and the task in which they aim to perform,
and this in turn is enabled by the motion matching,
which is at the core of the concept,
and in one fell swoop
the motion matching phase
acquires the pointer on an ad hoc basis,
selects the function that the user wishes to control,
dictates the body part or object to be used
in the interaction,
and defines the user's range of motion.
MatchPoint requires a simple, unmodified web camera.
It's low cost and highly deployable,
and the choice of input is free,
the user can choose the choice of input.
But other sensors could be used with the technique,
so we could extend into three dimensions
using depth sensors,
or use initial measurement units
in mobile or wearable devices,
or head mounted displays.
An important point is that MatchPoint makes no assumptions
of the users or the objects.
So there's no training or calibration.
It's posture agnostic.
The user doesn't have to be in the scene,
as long as the motion they generate
is witnessed by the camera.
And this has big implications
as an accessibility device,
because most gesture techniques
assume the user has hands.
In this technique,
as long as the user can generate some form of motion,
they can use the system.
But it also opens up the question
of whether one should design for a specific input modality.
It's been shown in the past
that the hands outperform the head for pointing,
and if we design with all input modalities in mind,
we'd have to design to the lowest common denominator.
However, in future work,
if we added post hoc object recognition,
interfaces could be presented
depending on which input modality the user uses.
We use circular motion,
but any shape can be used for the motion matching.
It just depends how distinctive the motion is,
and how easy the users can replicate it.
The output device also doesn't have to be a display,
as long as it can display motion
that the user can synchronize with.
The feedback also doesn't have to be visual.
For example, with a radio,
movement can be fed back to the user through audio channels.
So, spontaneous spatial coupling
is a dynamic technique,
which appropriates any body part or object
as a pointing device,
leveraging the benefits of motion matching
and cursor based interaction,
and it's a flexible and spontaneous,
touchless input over a distance.
Input can now be uniquely tailored to the context,
depending on the way the user manipulates the interface,
and MatchPoint is highly deployable and flexible.
And we believe this opens
a new design space for exploration.
Thank you.
(clapping)
- [Man] Do you have a question?
(mumbles)
Sorry, so how much computing power do you need?
Does it run on a really small device,
so that the matching, or does it
depend on the complexity of the scene?
- It depends on the complexity of the scene.
So at the moment we run it on desktop computers or surfaces,
but we're currently developing
mobile applications.
The feature detection is the limiting factor
in the CPU,
but we could also optimize the system using the GPU, so.
- [Questioner] Hi, so I'm Andy Wilson, from Microsoft.
So, that's nice work, I really like it.
I was curious if you had any idea
how to answer the question of,
if you have lots of motion in the scene,
how reliable is this system?
Now there's lots of different
aspects of that question that are sort of
frustratingly vague,
and I was wondering if you had an idea
about how to even approach evaluating that, the liability?
- So the system is built upon TraceMatch
which was introduced last year at UbiComp,
and we just published a user study
at this year's UbiComp.
And in that we looked at trying to avoid false positives.
However, we only did it with a limited number of users,
with a limited number of tasks.
So for example we didn't use a crowded scene.
But we designed this system according to
the tradeoff between making it
open enough so that users can accurately
match the motion,
while it's constrained enough
so that we avoid the false positives.
In terms of a crowded scene,
we'd need to look into that.
- [Questioner] Okay, thank you.
- Thank you.
- (mumbles) I have a follow up,
and somebody else can also think of a question.
So the complexity for the user.
So I think I really get sort of the second part
once I have locked to the motion,
that's really neat.
Do you have any experience on the first part?
How easy is it for people
to really get the concept of that they have to mimic,
with their interaction, a movement on the screen?
- So once again, referring back to the TraceMatch work,
the users seem to acquire the capability very quickly.
Depending on the input modality,
really depends on how well they can match the motion.
So everybody using hands and objects
performed brilliantly.
Head motions were a different story.
So we looked at two different speeds,
and for fast head movements,
a lot of users performed very badly.
And in terms of how
spontaneously people can pick this up,
there's a work called PathSync,
which is used to connect for motion matching,
and they deployed this as a public display,
at which users would just walk up and interact
and they had no problems.
- [Man] Okay, thank you.
There are no further questions,
let's thank the speaker again.