Unknown Speaker 0:00 Hi, my name is Vinay Krishna Sharma I will
be presenting our work on eye-gaze controlled robotic arm for persons with speech and motor
impairment. This work is done under the supervision of Dr. Pradipta Biswas in the intelligent inclusive
Interaction Design Lab at CPDM in IISc Bangalore, India. The lab members belong to all different backgrounds.
I personally work on human robot interaction, which is the study of robotic systems which
are designed or developed to be used by humans. It is also known by other names like man machine
interaction, human machine interaction and human robot collaboration. In the I3D Lab,
we are making solutions for education and entertainment of users with speech and motor
impairment. Such persons face challenges in natural hand speech and gesture movements
So for communication, persons with SSMI use the Speech/pointing charts, like the ones you
see on the screen, where the trainer has to analyze the users eye-gaze based on what the user
wants to say maybe a yes and no work before he wants to drink water or not. Or maybe he
for if he wants to see a complete word, then we might have to look at the these clusters
where the user is looking at. To complement the shortcomings of persons with speech and
motor impairment. We wanted to deploy a human robot interaction system which can allow the
users to we can place to draw to manipulate some objects in the environment using just
their eye gaze. So we worked with Vidyasagar school in Chennai, and we did some user trials
on the users which belong to all different age groups, all different genders and at different
educational backgrounds. The important point was that they do not have any finger movement
or like head isolations or speech. So the only move, the most prominent mode of interaction
was eye-gaze. We wanted to analyze eye-gaze as a modality to interact with the robot to pick
and place something in the environment. We designed two user studies. First one was a pick and
drop task where the user had to use his or her eye gaze to pick and place shuttlecocks in
the robot environment. And the second one was the reachability study where the user
had to reach randomly assigned target from a starting point using just his
PB Sir 2:24 we have developed and is controlled to video
see through interface to control her robotic arm. The system found applications for people
with severe speech and motor impairment. Users with severe motor impairment could undertake
simple pick and drop tests using our interface. The robotic arm moves based on the saccadic
gaze movement, and they can dwell on a screen region for picking and placing an object.
VKS 2:55 The results of this study were quite fascinating
as all the 18 participants were able to complete the task successfully. And as far as the task
completion time was concerned, there was no significant difference found between the persons
with SSMI and able bodied persons. the video shows a person come trying to pick and place shuttlecocks
from desired locations to desired destination using justa, I guess. Next, the reachability
study where the user had to reach to a ... randomly assigned a target from from the center
of the circuit. And the user can just use the four-way controls. So the user can move
the robot in just four perpendicular directions, like up and down left and right. And this,
the results were also like quite motivating as all 12 participants were able to complete
the task successfully. And we also noted that in the second session, the performance of
the persons with ssmi improved significantly.
PB Sir 3:57 our end users with severe motor impairment
are already involved with fabric printing activities. With the help of an able-bodied
person. Our eye-gaze controlled interface can enable the users to undertake such a job independently.
They can move the robotic arm anywhere in its working on Hill up through I guess and
can make a print by toiling at the middle icon up the screen. Our user study found that
they can bring the arm at any precise location using our interface.
VKS 4:27 This video shows an actual severe speech and
motor-impaired user using our proposed system to move a robotic arm from a randomly assigned
starting point to a designated target using the four way control displayed on the augmented
user interface on the surface pro tablet and he completes it very nicely. In both the previous
or user studies we have used a commercially available off the shelf eye tracker which
is made by Toby which is a leading eye tracker manufacturer so But it is costly. It needs
separate hardware to install. And it is also low latency but like it, it cannot be deployed
as a mass as a mass solution. So we also worked at making a indigenious, webcam based eye-tracker.
And to make the webcam work as an eye tracker, we used open source software called open face,
which is a face tracker, which gives you head position head orientation and facial landmarks.
And these parameters were used to derive eye vectors for a user. And when he was asked
to look at one of these randomly assigned nine blocks on the screen. So with this, we
were able to achieve where the user, we were able to track the users eye-gaze on these
nine regions of the screen, and these were like good enough to control a robotic arm
In the four perpendicular directions. The proposed webcam-based eye-gaze control allows
the user to control the mouse using just eye-gaze, which can be used to browse the internet
or maybe type a message or SMS, or do a Google search.
PB Sir 6:08 We have developed a webcam-based eye-gze tracer,
using open face library. Using that with webcam-based eye-gaze tracker, users can
operate the robotic arm through eye-gaze without needing any dedicated hardware for eye-gaze tracking.
VKS 6:23 The proposed webcam based eye tracker was used
for a reachability study again, where we found that in the second session, the performance
of the persons with speech and motor impaired improved significantly.
Thank You. We want conlcude this with a video.
PB Sir 6:42 We have also integrated computer vision
technology with the interface. In this part of the video, the robotic arm can be seen
to follow a blue object. The same algorithm can be configured for any particular color
or shape of object, or a set of markers within the robots working envelope.
PB Sir 7:05 The same eye-gaze controlled augmented reality display
can be extended to other cyber physical systems beyond robotics, like unmanned ground and
aerial vehicles. We can drive a car simply by looking at appropriate waypoints in a live
video feed or can run it in semi autonomous mode towards the target. In the last case
eye-gaze can be used to select one of many targets and the vehicle moves towards the target using
computer vision technology. We have used similar principle for unmanned aerial vehicles to
learning to operate a single interface and interaction technique. Users can operate multiple
cyber physical systems