Hi, I'm Iain Carson, a Ph.D. candidate at the University of St. Andrews, I'm just about to submit my thesis.
So if you find my work interesting, and would like to discuss opportunities, please do get in touch.
So today I'm going to be talking about the work we've done,
investigating the role of sensory concurrency as a feedback mechanism when learning how to use haptic spatiotemporal signals.
Our work shows that the visual information available during training has a big impact on how these signals are learnt.
The signals we're interested in exploring and interpreting are associated with
the branch of multimodal interfaces which are engaged with through "enactivity".
They feature continuous sensory outputs with implicit meaning, which reacts to dynamically to user action and the environment.
These can be contrasted with the cognitive interfaces we are perhaps more familiar with, such as most computers or this iPad,
which feature symbolic representations of information for cognitive interpretation and explicit
extraction and utilisation. Enactive interfaces have existed for many years in the form of,
for example, sensory substitution and augmentation devices which attempt to modify the way that we perceive the world around us.
Over the past decade,
we've seen more enactive interfaces appearing in HCI and demanding embodied applications such as virtual reality and driving. This example,
"The Enactive Torch" by Froese et. al. from 2012 is one of the first and simplest enactive devices.
When held, it vibrates at an intensity proportionate to the distance to the object
it's pointed towards. Users can sweep the device much like a blind man's cane,
and through such an interaction, can extract information about the environment and use the device for navigation.
But this information is not intuitive. It has to be learnt somehow.
Having straightforward mappings which are relatable and simple to comprehend helps with initial interpretation.
But there is a deeper learning process which comes with experience. Mastery results in fluent action and natural navigation.
We are interested in the mechanisms by which we can promote and enhance learning to mastery.
The role our other senses play in learning these new, vibrotactile signals is unclear.
In some situations, it's argued that sensory deprivation,
for example blindfolding, may force attention towards the novel signal aiding learning. In others,
users familiar with the task may be able to generalise their skills to understand new signals,
drawing parallels from past experience. And finally, by allowing users to keep their eyes open when learning new signals,
sight provides a valuable source of calibration, helping users understand the signal's
meaning. We hypothesised that a hybrid condition leveraging all of these
mechanisms may be worth exploring and set up a simple experiment to investigate.
We decided to use the vibrotactile output from the trigger motors in an Xbox One controller.
That's numbers one and two in the figure here.
to see how visual information presented during test performance could help or hinder the learning of a new haptic signal.
We encoded distance information much like The Enactive Torch into the virbtotactile intensity output by the left and right trigger.
Then we created a virtual maze system in which users would try to navigate using visual or haptic feedback or a combination of the two,
depending on conditions. Our conditions reflected the aforementioned learning mechanisms.
In our sensory deprivation condition, participants never saw the maze.
They were expected to learn to navigate the maze with only feedback on how far they had traversed.
In our skills transfer condition, the participants first learnt to traverse the maze using visual feedback, before attempting the maze using only haptic feedback.
And in our sensory concurrency condition,
we presented users with both full visual and full haptic feedback before the haptic only navigation was attempted.
Finally, we created a hybrid mechanism where we reduced the amount of visual information available,
so the haptic signal became the most useful source of navigation information while the visual information was still available for reference.
We ran twenty four participants,
six in each condition through three sessions at least a day apart, before testing them all in their ability to navigate the maze using just haptic feedback, as shown here.
In a mixed method approach, we collected various quantitative and qualitative metrics to determine the effectiveness of our conditions and the experience of participants during training.
We ran a generalised, linear, mixed model on the distance,
expressed as a number of blocks that participants traversed in each trial, and on the ratio of blocks traverse to walk collisions,
which we called progress.
We found that participants trained under our hybrid condition, travelled almost twice as far through the maze as other participants.
They also had high progress scores, suggesting that they were also competent at avoiding walls and taking efficient path through the mazes.
Plotting the distance against progress, we see some grouping between the conditions.
Participants in the Sensory Deprivation condition primarily fall into Quadrant one, showing cautious progress and poor distance.
The Skills Transfer and Sensory Concurrency groups fall into Quadrant three with weak overall performance,
but the Hybrid case accounts for most of the top distance, top progress trials in Quadrant two.
Sampling a few of these, we also found that the conditions had a key influence on strategy and behaviour of participants during the test session.
As can be seen from these path traces, participants in different conditions and with different block and progress scores utilised
different strategies to navigate the final mazes, indicating different levels of task and signal comprehension and comfort.
The sentiments were further reflected in Likert scale questionnaires and a NASA TLX score,
where participants in the hybrid condition reported the greatest understanding of the maze and game rules,
as well as agreeing that the first three sessions help them prepare for the final session.
So to conclude, we note the sensory deprivation does promote careful progress.
But it may take a long time for strong proficiency and task understanding to develop.
Sensory concurrency and Skills Transfer was similarly effective in teaching the new signal, but neither was as effective as the Hybrid condition.
When training participants to learn new and active signals,
we encouraged the presentation of conditions which allow expression of intent in a familiar context,
focussing attention through incentivising the value of the new signal and using
complimentary sensory information to provide feedback and maintain orientation.
Thanks very much.