- Fakourfar who did this work
about stabilized annotations for mobile remote assistance
with colleagues from the University of Calgary
and University of New Brunswick.
- Good morning, everyone.
I am Omid from University of Calgary,
and I'm gonna present my Masters thesis work,
which has been a work in collaboration
with University of New Brunswick.
So maybe you have been in this situation before.
Your car is broken down,
and if you are like me,
you have no idea how to fix it on your own.
We all might refer to the owner's manual,
or I would even try using YouTube's
do-it-yourself videos,
but the problem is we are often facing problems
in which the solution is not clear,
or we need constant guidance or feedback
from an expert helper.
So if I use the manual
or a YouTube video,
I won't have the right support,
and I probably won't be able to fix my car
on my own.
But wait, we all have a good handyman friend
from hometown who might be able to help.
So the question is how?
Well, we can set up a video call
to ask for help,
and we have mobile video conferencing
that could allow us to communicate
with each other,
and we can choose between a variety
of devices to make this communication happen.
So, your friend can use a tablet
or any other handheld device, like a smart phone,
to communicate with you,
and if you need both hands to work on the task,
you can use a head-mounted display,
but they are not very common yet,
or some hand held devices,
like tablets or smart phones
that are widely available, cheap,
and it's easy to make applications for them.
But wait, so I don't know what my friend
is talking about,
so this looks actually complicated,
but we know from the previous work
that being able to annotate the work space
by the helper person is critical in many scenarios.
So the helper needs to be able
to reach into the scene and somehow
annotate the workspace.
So there are also lots of technologies in this area,
like annotations which are atop the video feed
versus those that are anchored to the objects
in the scene.
We call the first type non-stabilized annotations.
So these annotations are drawn atop the video
and actually move with the camera movement.
So as you can see, when I move the camera,
at this point, they have lost their intended meaning.
The second type of annotations are stabilized annotations,
which are anchored to the objects
on which they were drawn upon.
So as you can see,
when I move the camera.
The annotations are still in their own place,
and when I move the camera back again,
I can still use them.
But this type of annotation needs more effort
and cost to be implemented,
so it need augmented reality dragging.
It needs AR image targets,
like these that I have here,
but as a driver,
who has got any of these laying around
in their car?
So the question is,
are they really worth the extra cost and effort?
So as I said,
there are commodity technologies that seem
to support remote assistance scenarios.
They are really cool and even popular
among the users,
but the question is,
are they really worth the cost and effort?
So it looks like they're actually not.
So actually they are not the best option
for all the scenarios.
Instead there are cheaper and easier approaches
that seem to be just enough for many tasks,
and they are easier to implement.
So the bigger question is,
how should we design effective annotation systems
for mobile video conferencing
with all these different devices
and annotation types.
Let me first show you these technologies again.
So in terms of the device configuration
for the worker,
we have tablets that are cheaper,
and they provide more mobility for the worker person,
and we have head-mounted displays
that free both hands,
and they align the camera view with the eyes view.
And in terms of the annotation systems,
let's watch them again in action,
but this time, in the context of our study tasks.
So we have non-stabilized annotations
that, as you can see, are drawn atop the video
and not anchored to any objects in the scene,
and stabilized annotations
that require augmented reality techniques to be implemented,
but they are anchored to the objects in the workspace.
So how should we choose a combination
among these device configurations
and annotation systems?
So actually, prior work have already studied
most of these,
but there are two main problems
when you read through them.
The first is that they have not been very conclusive,
and the more important one is that
each study was run differently.
By that I mean they have each chosen different task factors,
and these task factors can lead different technologies
to perform differently,
and now I'm gonna tell you about the task factors
that we are identified and use in our work.
So we have locus of knowledge,
with all the knowledge for completing the task
is located at the helper side,
or it's shared between the helper and the worker.
We had movement and size of the workspace
and the physical complexity of the task.
So based on the history of research in this area,
we were able to come up with four tasks,
and we evaluated lots of task factors in all of them.
So like if it's a two D or three D task,
if it needs both hands,
or it can be accomplished with just one hand,
and these are our four tasks.
So we had a tangram puzzle test,
which is a construction task.
It can be accomplished with just one hand,
and it has a planar work space.
A graph task, which has a larger work area,
the knowledge is shared,
and it's again, a planar.
Oh, sorry.
It has a planar workspace,
pretty much like the car repair task.
An origami task,
which needs both hands to work,
and it's a three D task,
and a Lego repair task,
in which the knowledge is again shared,
and it needs both hands to work.
So we tried to address the problem
with previous studies
and consider all these task factors
by designing and running two different user studies.
In the first one, we compared stabilized annotations
with non-stabilized,
and head-mounted displays with tablets,
and in the second one,
we studied another annotation system
that we thought might be just enough
for many tasks,
and compared them with non-stabilized annotations.
So the first study,
so in this one, we recruited 16 pairs of participants,
and we designed it within subjects study,
and each pair went through
all the four possible configurations.
And this is how we set up our study.
So the helper and worker
were actually in the same room
to get rid of the audio communication issues,
but the helper used a tablet
to communicate with the worker
and to draw annotations,
and the worker used a tablet
for two tasks and a head-mounted display for the other two,
and our findings.
So here are some quick charts
about utility of stabilized annotations.
So it turns out that a stabilized annotations
were no faster than non-stabilized version
in terms of task completion time.
That was interesting for us,
but they were actually preferred by the users.
So, we don't have a good reason for that,
but we just think that
might be an effect of novelty.
And here is another important finding
that we had from observing our participants
during the tasks.
So annotating video is really hard.
I'm gonna show you a video that will show you
a problem that we saw several times
with different groups.
In this video, you will see the helper
trying to draw an annotation on the video,
but at the same time,
the worker is moving around the object,
which is an origami paper,
and this frustrates the helper.
Let's watch the video,
and I'm gonna get back to this point later on.
- [Woman] So basically, you wanna like fold.
I'm gonna draw on this, okay?
Fold like that,
like a square.
- [Woman] Do I fold the whole thing?
- So, maybe the most interesting finding was this one.
So we were able to classify the annotations
into three main categories,
reference, procedural, and pointers.
So, the first type was reference annotations
that were needed to be referred later on,
like end states or legends.
So in this video,
you can see the tangram puzzle task,
the helper draws the outline of the final shape,
and then the worker works
on his own to do the task.
Then we had the procedural annotations
that show some kind of process,
like folding or rotating objects,
and this is one from the origami task.
And we had pointer annotations
that were used to temporarily point
towards objects in the scene.
So this one is from the Lego repair task,
and this is the distribution
of all these types of annotations
across the tasks.
So as you can see,
a large portion of annotations
are either procedural or pointers,
about 61%,
which means they are needed only for a short
amount of time, maybe a few seconds.
So to summarize, stabilized annotations were preferred
without having any actual time benefit,
and after classifying the annotations,
we found out that most annotations are ephemeral,
and annotating video is hard,
regardless of whether the annotations
are stabilized or non-stabilized.
But if annotating video is hard,
now you can guess that annotating pictures is easy.
So as I told you before,
most of annotations are ephemeral,
so what if we could make temporal annotations,
but not on live video,
but on still frames, on pictures?
So we call such technique,
temporarily stabilized annotations.
So I'm gonna show you a video.
So here the helper start drawing the annotations.
The screen freezes,
and it keeps frozen for two seconds,
and after two seconds,
it just continues the video feed.
This technique is also easier to implement
than the stabilized version.
So this led us to this question
for another study on another annotation system.
We called it F two or freeze-frame,
and that is, does the simply freeze-frame approach
for annotating provide any benefit
over annotating the live video?
So for this study,
we recruited six pairs of participants.
They only use a tablet,
both the helper and the worker,
and we used the same Lego and origami task,
and again, we had a within subjects study.
So in one sentence, it worked pretty well.
The good thing was that now the helper
could draw on a still frame,
a picture instead of live video,
so it was much easier for them,
but those few seconds of freezing the frame
caused some loss of context for the helper
because they had no clue what's going on
on the other side.
So as you can see in this video,
the helper starts drawing the annotation to the right.
The screen is frozen,
but after that, the worker has already
rotated the structure.
So let's talk about the pros and cons
of this technique.
First of all, since we are annotating on a still frames,
it's much easier,
and we won't have the problem with moving camera or objects.
It's also easier to implement
because it does not any sort of
augmented reality technologies,
but you saw the loss of context from them
which can hinder the real time feedback.
But we think this problem is easily addressed.
We can provide another live video for the helper
as he is drawing the annotations
at the corner of the screen.
So temporal annotations are a very easy technique
that can support most with assistance scenarios,
and this what we are going to do
in our future work.
So to summarize, we identified three types of annotations,
of which the procedural and pointed annotations
are only needed for a short time,
and most tasks can actually be accomplished
with these ephemeral annotations.
So we looked into annotating pictures
instead of live video,
and they would actually win.
So that's our take away from this paper,
and thank you.
(audience applauding)
- Alright, we have time for some questions.
You can line up at the microphone for any questions.
In the meantime, I'll ask you one.
I was curious,
can you imagine what kind of scenarios
would these like mobile annotations
be useful compared to like
using self-help videos on YouTube,
which I think a lot of people use right now
to help themselves fix a car?
- That's good question.
So, we think they are used mainly for tasks
that need some constant guidance
or feedback from the helper,
so like, the car repair task,
it might not be too risky,
but there are maybe more riskier tasks,
like repairing a jet engine or something
that they actually need constant feedback
to just don't do something wrong,
and then that's why you need to just
refer to objects and point at things.
So that's what I think that.
- [Man] (mumbles) University of Illinois in Chicago,
in a temporal stabilization model,
do you freeze the video on both ends,
or is it just the helper's video
that's being frozen?
- It's on both ends.
- [Man] Okay, thanks.
(murmuring)
- [Man] Did you consider fading the annotations
instead of freezing the video?
- What do you mean by fading?
- [Man] Like, they appear,
and then after two seconds,
they gradually fade out.
- So, that's a good approach.
So it's basically now both freezing and fading,
so it fades after the two seconds,
but what we needed to do was to have
some sort of stabilization.
So we get to that stabilization,
that freezing the frame,
but with just fading,
so you don't have the annotations there
for any amount of time.
So you can't stabilize them in somewhere.
- [Man] I was just curious if the frame was frozen
for a fixed amount of time for like two seconds,
or as long as it took to annotate the frame?
- So it was fixed,
so when you actually draw that annotations,
and you put your finger up,
then you had two seconds for annotations to be there,
and then it was unfrozen.
So it was exactly two seconds.
- [Man] If you let it fade away,
it could give the user the opportunity
by just clicking to let it stay for another while.
Wouldn't that be an option?
- That's right.
So that's what some prior works have worked on,
so you give the control of whether you want
the annotations to be there for a few seconds
or you want to automatically fade them
or unfroze them.
So there are some approaches on this,
but one of them is absolutely that you have the control
and there to fade them or have them on the screen.
(murmuring)
(audience applauding)