Hello, My name is Roger
I am from the Augmented Human Lab
at the Univerity of Auckland, and today, I will be talking about AISee, a wearable assistive
device to support visually impaired grocery shoppers.
This work was done in collaboration
with my colleagues Denys Matthies, Haimo Zhang, and Prof. Suranga Nanayakkara
together with the support of all the members of the Augmented Human Lab.
AISee is an assistive bone conduction
headset that enables users to access to visual information by pointing at it.
This technology adds to the previous body
of work on wearable assistive devices for people that has visual impairments
in specific to the domain of object identification and grocery shopping.
Some of the existing work in this domain are
Vizwiz, where users are able to acess to a crowdsourcing platform
by taking a photo with their mobile phones. Or FingerReader2.0,
which was presented here at IMWUT back in 2018, and enabled users to access
to visual information through a finger-worn camera.
We can also find existing commercially available devices such as Orcam or
Microsoft SeingAI, which are currently used in the real world.
In this work, we follow a human-centered
iterative design process and tailor an apparatus to the needs of our target group.
We provide an understanding of the user
needs via user journey and interviews together with detailed insights on our
technical design decisions from our iterative design process.
Finally, we provide the results of an empirical evaluation and identify
which users could benefit the most from our apparatus.
Through the user journeys, we identified grocery shopping and item
identification as key challenges for our target group,
where all the participants reported facing difficulties with food-related items
such as cooking, shopping, or understanding food condition
The interviews pinpoint the head as
the preferred location to place wearable assistive technology
Users reported wearing headphones daily.
In specific four participants reported using bone-conduction headphones.
These headphones do not block the ear canal,
and Interestingly, users already identify them as assistive technology.
They use these, together with their phones for navigation purposes, playing music,
calls, etc, Following the user preferences and insights,
we tailored a headset device to the user's needs.
Multiple iterations were conducted to develop and tune the device.
For this presentation,
I will focus on describing the final iteration. On the paper,
you can find further details on the other iterations.
The end device has four main components, a
ring that acts as a trigger to start the recognition.
A camera mounted on the headphones
that captures the user view, an audio transducer that delivers
the information without blocking the ear canal, and finally, a processing unit.
For the trigger.
we designed a custom-made flexible PCB with a shape of a ring.
This connects to the headset through
Bluetooth low energy and has a physical push button.
The ring is powered by a small curved battery and last for multiple days.
The processing unit integrates a system
on board module that runs Android OS together with a 13 Mpx camera.
Onboard, we
perform preprocessing and conditioning of the image, as well as run a text
to speech engine. The deep learning algorithms are run on the cloud.
Here is a diagram of how the different
parts are integrated into the headset at a more conceptual level.
From a functional point of view,
when the user triggers the ring, a command is sent to the headset.
This one wakes up, captures an image, and sends this to the cloud.
In the cloud, we use a Single-Shot Detection model
to locate the finger and crop the region of interest.
Then the cropped image is sent to an Ai
cloud service to extract generic features such as text, logos, or annotations.
Based on the results,
we then generate a sentence and deliver this to the user.
We evaluated the apparatus in a controlled
environment, as well as conducted pilot experiments in the wild.
This short clip was extracted from the controlled experiment,
where users were given a product and ask to provide certain information
Following the visual categorization provided by the World Health Organization,
participants belonging to category four and category three,
which are those who have a severe visual impairment but still have certain
eyesight, are the ones that could benefit the most from our apparatus.
These are likely to identify where the information is.
However, they require the device to access to it.
These results were reflected
on the performance of the apparatus, as well as its usability.
As future work,
we are looking forward to further iterate
the device by incorporating the feedback provided by the users and design a more
ergonomic apparatus with the goal of a wear and forget user experience.
We also would like to conduct a more
extensive user study with participants in category 3 and 4.
finally, we would like to compare our device with state of the art technology
for more details, please check out our paper.