Today, I will present our work, Video-Annotated Augumented Reality Assembly Totorials.
Assembly and disassembly procedures are recurring activities in today's industrialized society.
Traditional video tutorials can provide information on how to build or maintain complex objects.
However, following video tutorials may be challenging,
when the user's current viewpoint is not aligned with the one in the video.
On the other hand, due to their spatial anchoring, assembly instructions are ideally suited for AR applications.
However, the state-of-the-art methods are limited to certain object types or classes,
and do not extract undetectable information such as small parts and manipulations to the assembly objects.
To this end, we make four contributions to this field.
We introduce a novel approach for extracting complete 3D assembly instructions
from a video tutorial utilizing the object’s assembly graph to infer otherwise undetectable assembly steps.
We introduce a novel visualization technique for assembly instructions combining 3D animations
with video information to encompass all information from input video tutorials.
Video annotations provide information on otherwise undetectable manipulations.
We introduce a practical approach for an AR tutorial setup using commonly available hardware
by utilizing a smart-phone for object tracking and a monitor as AR mirror.
We demonstrate the completeness of our extraction and visualization approach
by performing a user study comparing against typical video tutorials.
Our system can be explained in three steps.
Input to our system is a video tutorial and a 3D model of the object.
Our system animates each of the parts following the assembly procedure that is demonstrated in the input video.
First, we compute the assembly graph from the 3D model.
We use the assembly graph to support the detection of the removal of assembly parts in the video.
To identify unblocked parts that can be removed, we compute blocking relations between all parts of the 3D model.
We extract the assembly sequence from the video input in the second step.
Finally, we generate step by step animation in the third step.
We implemented three types of video label placement based on users’ feedback.
The first prototype placed the video label on a 3D plane connected to the object.
A cone indicated the original camera position.
We assumed that the moving 3D label would not distract the users.
However, the actual feedback from the users in a pilot study was opposite.
In the next iteration we fixed the video label at the corner of the display,
but the position was criticized as being too far from the object.
In the last iteration, we placed the label centered above the object to bring the whole content to the user's view.
We validated the results of our approach to automatically create
step-by-step AR instructions from an unmodified video input.
We compared the extracted AR instructions to traditional video tutorials.
In our study, participants were asked to assemble a car model with 34 parts,
attached to a wireless IMU for tracking.
After assembling half of the model using one method,
participants continued assembling the other half using the other method.
We collected these measurements to verify the following six hypotheses, suggesting the superiority of our approach.
Statistical analysis revealed that we can accept three hypotheses fully and one partly.
There are four design recommendations from our findings.
First. One of the keys to reduce the mental effort is step-by-step animated instructions.
It allows users to select an acceptable speed to observe the instructions.
Second. Another key is to allow users to control the viewpoint to align the visuals of real-world object with ones in the AR mirror.
Third. Even for thin and non-rigid objects, which are hard to detect with computer vision algorithms,
we can still use video label to salvage the information.
And Finally, fetching visual information directly from input videos can align the visuals of real-world object
and virtual ones and this facilitates better visual search.
Thank you for your attension.