Welcome to my talk „Augmented Reality Training for Industrial Assembly Work”.
My name is Sebastian Büttner. In this talk, I’d like to address the question
“Are Projection-based Augmented Reality Assistive System an Appropriate Tool for Assembly Training?”.
This work was carried out by me and my colleagues Michael Prilla and Carsten Röcker.
When we look into industrial production, obviously
a lot of things have changed over the last decades.
Besides automation, we observe that complex products are still assembled in a manual way.
In the past, various projection-based AR systems have been presented to support workers.
These systems use projectors to show work instructions in the work area.
Some authors also use the terms in-situ projections or projection mapping.
Previous work has been motivated by different scenarios:
The first scenario refers to the use of such systems for a flexible production.
The second scenario is the use of such systems to support people with disabilities.
We call these first two scenarios permanent-use scenario since the assistive system is used
permanently during work. With our work, we aim to look into a third
scenario – the training scenario. We’d like to know whether such systems can
be used for on-the-job training for new employees. Consequently, with our work we focus on the
efficiency of building up procedural knowledge and on knowledge sustainability.
So the goal of our work is to learn about the training efficiency and the knowledge
sustainability of AR assistive systems. For this purpose, we ran an experiment.
We compared AR training with two control groups: The first control group is a personal training
in which a trainer shows the assembly process to the trainee.
We consider this control group as an optimal-case scenario.
The second control group is self-learning with technical documentation only.
We consider this second control group as a low-cost case, since no personnel is required.
Of course, this is not the first work dealing with AR support for procedural tasks.
At CHI 2003 Tang et al. presented work on AR support for object assembly.
They compared the AR-supported assembly with different computer instructions and a paper manual.
In their study, they were able to show that
AR support reduced the error rates of the task.
When it comes to in-situ projections, Funk et al. presented various studies on the potential
of projection-based AR. They showed that locating parts and assembly
actions are completed faster compared to other instruction types.
As many other studies, the mentioned works focus on task completion time, error rate
and perceived work load measured by the NASA-TLX. In 2019, Wilschut et al. presented a first
work about learning with projection-based AR.
They measured the immediate learning success after a training session with projection-based AR.
However, they could not find any differences
compared to working with electronic work instructions. According to our knowledge, there is no previous
study about the long-term effects of projection-based AR training.
Consequently, there is only weak empirical evidence regarding the training efficiency
of the system and no evidence regarding the knowledge sustainability.
As already mentioned, we want to consider two effects:
First, we look into training efficiency, which we consider as the time taken for learning
a procedural task. We assumed that a personal training is the
optimal case in terms of efficiency and consequently better than paper-based training or AR training.
But we assumed that AR training is still more efficient than a paper-based training.
These relations are reflected by our hypotheses H1 to H3 which you can find in our paper.
Second, we consider knowledge sustainability, by which we mean how well participants remember
the task after one week. Here, we assumed that users could benefit
from the spatial presentation of information in the personal training but also in the AR training.
Consequently, we assumed that personal training
and AR training would lead to a better knowledge sustainability than a paper-based training.
However, we did not assume any difference in the AR training and the personal training.
The hypothesis H4 to H6 in the paper relate to the knowledge sustainability.
So, here’s a short overview on our study design:
We ran a study with a between-subject design with groups for the three different training conditions:
Personal training, AR training and paper-based training.
We recruited 33 participants.
However, 6 participants dropped out during the study and 3 participants have been excluded
from the data analysis through incomplete learning in the first session.
From the remaining participants, 20 were male and 4 female.
The age ranged from 14 to 51 with a mean of 25 years.
The experiment was carried out in German and 18 were German natives.
The non-native speakers were balanced to the groups.
Each of the participants took part in 4 sessions:
A pre-test session, two training sessions and an evaluation session.
So, let’s go more into the details of the procedure: In the first session, we did some pre-tests
and collected the demographic data. Based on the data of this session, we assigned
the participants to one of the three groups. In the first training session, the participants
were trained in the assembly task. We let the participants execute the assembly
task repeatedly under the training condition. We stopped the learning phase when they were
able to execute the task without help and without error.
We measured the number of executions and the time.
After 24 hours, we scheduled the second training session.
At the beginning of the second training session, we let the participants execute the assembly
case without any help. We observed the task and noted any errors.
If participants made errors, which was usually the case, we continued with a second training session.
As on the first training day, we let the participants
execute the task under the training condition, until they again could do it without help
and errors. Again, we measured time and the number of
executions. We invited the participants to an evaluation
session 6 days after the second training. In this session, we evaluated the remaining
knowledge about the assembly task be two means: First, we let the participants execute the
task again without help and observed errors. Second, we integrated a knowledge quiz.
Using these two methods, we were able to learn about the remaining knowledge after one week.
So here’s a short overview on our setup that was used for all three conditions.
You can see an assembly workplace, which is similar to those that you can find in industry.
It basically consists of a table and some grab containers for the required parts.
Above the table is the projector for the in-situ projections.
Furthermore, we attached a camera to record all sessions with the consent of the participants.
We attached a base plate to the table on which all parts of the task had to be mounted.
While previous studies often used Lego Duplo bricks, we decided to use a different assembly task.
We used this pneumatic construction set by the manufacturer Fischertechnik
that you can see in the image, which has 24 parts in total.
Since the base plate was fixed on the table, 23 steps had to be executed in a defined sequence.
Compared to Lego tasks, this task has different kinds of assembly steps, which makes it a
little more complex. Still, no tools are needed for this task.
Here you can see an overview of our assistive system.
The system mimics previous research and commercial systems.
It shows projected instructions in the work area, such as videos, mounting points, picking
information at the grab containers or error indications.
We implemented four different support levels with different amounts of support.
The participants were free to change the support level during the experiment whenever they
wanted. For the recognition of picking actions and
correct or incorrect assembly steps, we used a Wizard-of-Oz approach.
And here you can see the assistive system in action.
This video was taken during the study. It shows a participant working with the system.
This is the second level of support, so the participant did get information about mounting
points and the location of parts. Now you can see that the participant made
an error, so the system makes the user aware of the error.
Now the error is fixed and the participant can continue the assembly process.
The manuals for the other two conditions are straight-forward.
Of course, we tried to present consistent information in all three groups as far as
possible. In the image you can see the paper manual.
For the trainer, we defined a short guideline. The trainer was always the same person and
he followed a defined standardized procedure. As I mentioned earlier, we evaluated the knowledge
of the participants after a week not only by executing the task.
We also included a quiz that each of the participants had to complete on a computer screen.
This quiz contained 25 questions. There were two question types, as you can
see in the photos. Now, let’s get to the results.
First, we look into the results of the training efficiency.
To the left, you can see the number of training cycles.
To the right, you can see the total training time in minutes.
One interesting finding is that we did not have any significant differences between the
groups on the first training day. However, on the second training day, the participants
of the personal training group needed much fewer training cycles compared to the other
groups. The number of training cycles of this group
is below 1, which means that a lot of participants that received the personal training did not
need the second training because they executed the task without any errors after 24 hours.
So what we conclude from this data is that the personal training group had a more comprehensive
understanding of the task after the first training day.
Again, I’d like to stress that the termination condition on the first training day was identical
in all groups. Furthermore, there were no significant differences
in the training sequences or times on the first day.
Still, the personal training seems to have had the best effect on the first day, resulting
in a shorter training on the second day. Comparing the AR group to the paper manual
group, we did not see significant differences. However, I have to add that we excluded two
participants from the paper group that systematically learned the task in an incorrect way.
This was only possible in the paper group, since participants in that group did not receive
any feedback. If we hadn’t excluded these two data sets,
the paper condition would have resulted in worse training efficiency.
So, we can clearly say that AR assistive systems prevent the systematical learning of incorrect
work steps, which indirectly leads to a better learning efficiency.
When we look into the data of the knowledge sustainability, we can see a significant difference
of errors between the personal training group and the other two groups after 24 hours.
The average number of errors in the personal training group is below one, meaning that
a lot of participants did not produce any errors after 24 hours.
However, when we look into the number of errors when executing the task after one week, there
are no significant differences between the groups.
The same holds true for the quiz results. On average, the participants answered about
20 out of 25 questions correctly, regardless of the group.
So what does that mean? Well, even though we had the small difference
after 24 hours, we did not recognize any differences in the knowledge sustainability after one
week. Not in the quality of the replication of the
task. Not in the quiz.
And even if we broke down the results based on error or question categories, we could
not see any differences. You can find more details on that in the paper.
So basically, once the task was learned, the knowledge persisted in the same way, independent
from the learning condition. So last, we were questioning why there were
differences after 24 hours, but not after one week.
It might be the case that our study setup with the second learning day balanced out the differences.
Let’s look back at our initial hypothesis:
With regard to training efficiency, our study showed a slight advantage of the personal
training over the other two conditions. However, we have to reject the hypothesis
that AR-based training is faster than self-learning with a paper manual.
Regarding the training sustainability, we could not measure any differences between the groups.
Consequently, we have to reject the hypothesis
that spatial representation of information leads to a better knowledge retention.
So, I come to the conclusion of my talk: We ran an experiment that investigated projection-based
AR assistive systems for training. As expected, we found that a personal training
had the best training efficiency. AR-based training can be used as trainer-less
alternative. It successfully prevents systematic incorrect
learning. However, other advantages could not be proven
by our study. We found that once the assembly task is properly
trained, the recall precision is independent from the training condition under which the
task was learned. As future work, we plan to investigate how
certain aspects from personal training can be integrated into assistive systems.
We believe that if assistive systems become more intelligent, the qualities of a personal training might be reached.
Thanks for listening to my talk.
If you have any questions or comments, please get in touch with us.
Thank you.