- Technical issues...
I hope I don't bother you with my technical issues.
Potentially, something happens.
Have you ever wondered how our brain is structured or wired?
Our brain is composed of billions of neurons and fibers,
and these neurons and fibers are responsible
for transmitting electrical signals among each other.
And neurons and fibers constitute a sort of cable salad
in nanometric scale.
Can you imagine tracing a single fire
in this cable salad?
According to legend, Alexander the Great
had a similar problem which is known as Gordian Knot,
and his solution to that problem
was just cutting the wire salad or the cable knot
by his sword.
Of course, neural scientists work more elegantly.
What they do is they take a small piece of brain,
a tissue, brain tissue,
and they cut this tissue into very thin slices,
and from each slice they take an electro-microscopic image.
And they end up with a 3-D image stack,
a volume of images.
After that, they need to annotate
the corresponding cross sections in each slice,
slice by slice,
and link these tabular objects.
This operation is mainly done
by semi-automated manual input,
and semi-automated annotation systems,
and the domain is called connectomics,
and it is sort of a brain-mapping process.
In the neuroscience literature
there are certain applications.
In the neuroscience domain there are certain applications
which are based on manual annotation.
Here we see Eyewire,
and on the right-hand side you can follow maybe
the manual input with the mouse cursor.
As you notice, from time to time,
the user has to pilot the system
to let the automated system know where the trace fiber,
the cross section is located.
In this talk, I will present you
not the manual annotation case,
but the gaze-input case.
Our system is called Eyetrace,
and I will also talk about our user experiment results.
But before that, what is the problematic?
What is the shortcoming of manual annotation?
In neuroscience domain,
people have done some series of research,
and Nakamura et al. has presented their results
in Nature 2013.
In this work, they have presented a successful trace
of 380 neurons,
and it took them 15,000 person-hours.
That means 20 people, around 20 people,
a lab full of people,
were working half a year long only for annotation task.
And this was only actually mapping not a human brain,
Actually it was a fruit fly.
And from human-computer interaction literature
actually we know the that manual annotation tasks
can be describe as a function of fiber size
and the distance of the fiber to the...
I'm sorry.
The target size and the location of the target
to the mouse pointer.
And several reserachers in the CHI domain
and the HCI domain
tells us that potentially gaze interaction,
gaze input can replaced,
and manual input can be replaced by gaze interaction.
So I just want to test whether you agree
with this hypothesis.
Can we replace manual input, manual interaction,
with gaze input?
Here we have a test data,
and I want you to just stare at,
select a particle, a cross-section,
something like this,
or this,
and then just try to follow as the simulation,
the animation progresses.
I'm sorry.
I need to...
Okay.
Were you able to follow the cross-sections
with your eyes only?
Maybe a few hands.
Okay, thank you.
We had this same intuition,
and with this motivation
we have implemented our Eye-trace system.
Our system makes use of several open-source libraries
and tools.
On the visualization side,
we make use of IMechE and Fiche,
and in the orchestration for experiments,
we've made use of the Miche,
which is an extension for the IMechE.
And in the interaction side,
we have that low-budget, low-cost eye checker,
which is Eyes Right.
And for the interaction as a second modality,
we make use of a game pad controller.
With the game pad controller,
the user was allowed to move and pen,
pen and zoom the visualization,
and also start and stop the tracing trials.
Here you see our system in action.
We had to use a chin rest, as you might have noticed,
because the accuracy of the eye checker
was not as high as we wanted to.
But it was a low-budget solution,
so for gamification purposes,
it was potentially actually ideal.
Our user experiment has investigated two main questions.
We wanted to know how the tracing accuracy
was effected in terms of tracing speed,
represented with this symbol,
and of course the size and morphology of the fibers
from large, medium, and small categories.
On the second side, the second part,
we were interested in how the repetition of,
how repetitive trials of tracing same fiber
was effecting the tracing accuracy.
In this category, we were only using
one particular tracing speed,
which was 20 frames per second.
But I will come these details later.
Prior to our experiments,
we have prepared a ground truth data
with manual annotation.
This has been performed by our expert neuroscientist
at five frames per second approximately,
and we have measured the accuracy
by means of (mumbles unintelligibly)
and comparison of the participant trials
with the ground truth data, of course.
Our fibers were then categorized,
the silicate fibers were then categorized
in three different morphologies
denoted by large, medium, and small.
And these are measured in pixels, of course.
In our experiment, 20 graduate and undergraduate students
have participated.
We had to discard one participant,
the results of one participant,
because it was definitely, obviously an alpha.
The details are in the paper.
And the experiment which I am presenting here
has composed of three main stages.
In the first stage, in the training phase,
the participants have given their consent first,
have asked to give their consent
for this experiment,
and they have been trained for different fiber morphologies
in these categories,
and then they were trained for using the game pad
for panning and zooming,
and starting and stopping the tracing.
And also they have been informed
about the eye-checking interaction,
and line-point eye-checking calibration
has been performed.
In the second stage,
which was actually the main part
of this experiment,
we have tried to answer the first question,
the effect of tracing speed and fiber morphology
on the tracing accuracy.
And in the last stage,
we were interested in the effect of repetition
on tracing accuracy.
So in the main experiment part,
the participants were asked to perform 48 trials.
This is actually a basic map.
It was the experiment in the...
Experiment only 12 different fibers, unique fibers,
were included from three different categories,
and we had 10, 20, 30, and 40 frames per second
as speed levels.
Of course, the list of trials were randomized
to avoid learning effect,
and at the end of each trial,
we asked the participants to rate the level of difficulty
of tracing that fiber from one to five
on a Likert scale.
One was very easy, and five was very difficult.
For the third part of the experiment
in the repetition session,
the participants were asked to do eight ordered trials.
So no randomization, of course.
In the first part, they have been asked
to trace the same small fiber four consecutive times.
And in the second part,
a medium fiber four consecutive times again.
And our tracing speed was 20 frames per second constant,
and the participants have been asked again
for rating the difficulty of tracing that,
making this task.
And our results looked like this.
In the speed-size-accuracy experiment,
for large fibers we were not able
to observe any significant effect
of working with large fibers.
On the tracing accuracy, there was no significant effect.
Here, you see actually the sample data set.
This is actually the boundaries of our fiber
which has to be traced.
I'm sorry. (clears throat)
And the dashed lines represent
the average participant performance
in 20 and 40 frames per second.
As you might have noticed,
the participants' results are remaining
within the boundaries of the large fibers,
represented in large fibers.
And in terms of medium and small fibers,
we have noticed,
we have observed significant difference,
as you might have noticed,
that the boundaries are actually exceeding
the representative fiber boundaries.
That means we had actually less accurate results.
So these fibers were more difficult to trace
in terms of accuracy.
The repetition tasks, the repetition experiment,
has shown that we have actually,
we were not able to observe any significant differences
in terms of tracing accuracy.
In terms of task difficulty,
we had some interesting results.
As you see on the left-hand side,
the difficulty ratings were proportional
to the tracing speed,
so the more speed we have,
the more difficult the tasks were perceived.
In terms of fiber size,
the small fibers were significantly more complex to trace.
And on the repetition side, on the right-hand side,
we had only observed one certain case
which was tracing small fibers.
There was a significant difference
in terms of difficulty ratings
between the first and fourth trial.
That means this, for small trials,
for small fibers,
the repetition might help to trace
a learning tracing actually.
At the end, yeah.
In conclusion, I tried to summarize the take-home messages.
So we tried to implement proof of concept type of test bed,
and our user study results
show that actually we had a certain accuracy.
We have other results in our paper
which are also significant to note.
I suggest you to read that.
But in summary, mouse was more accurate than gaze roughly,
but roughly speaking,
but we had actually,
we could achieve more speed in terms of
by using gaze interaction.
As we used a low-budget eye checker,
we are planning to use a better eye checker
to continue our research,
and we see potential of using our system
for supporting proofreading tasks
for current semi-automated image annotation pipelines.
Thank you.
(audience applauds)
- [Narrator] (mumbles unintelligibly)
of Texas State University, very nice work.
If I may suggestion,
so as you correctly metioned,
you're using Eye Tribe eye tracker
so it has some accuracy correct statistics,
so it's very interesting
to have maybe another objective point
if you're going to use more accurate eye tracker,
in terms of how much faster you're going to be.
Maybe you're going to be even faster and more accurate.
Because actually...
- That's our goal.
- [Narrator] Eye Tribe has some very challenges
in terms of accuracy.
- We observed that also. Thank you.
- [Narrator] Thank you.
- [Questioner] Hi. Nice talk.
Quick question,
so your mouse-based segmentation
was done by an expert, right?
- That's right.
- [Questioner] And the gazing was not done
by experts, right?
- The gaze tracing was done by participants.
- [Questioner] Okay, so maybe it would even be better
with an expert, right?
- Of course.
But we wanted to test with novice users,
and we wanted to see the potential of using this system
for maybe potentially for gamification aspects.
That's why we used novice users also.
But partially they were from the neuroscience domain,
so they were not totally unfamiliar
with what was going on.
- [Questioner] Yup. Thank you.
- Thank you.
(audience applauds)