Assembly and disassembly procedures are recurring activities in today's industrialized society.
Traditional video tutorials can provide information on how to build or maintain complex objects.
However, following video tutorials may be challenging, when the user's current viewpoint is not aligned with the one in the video.​
YouTube or Instructables, provide information on how to build or maintain complex objects.
Video-based tutorials have been proposed for AR tutorials that display 2D videos overlaid over the AR view to communicate human motion,
general task workflows related to real-world objects, or assembly instructions​.
On the other hand, due to their spatial anchoring, assembly instructions are ideally suited for AR applications.
However, the state-of-the-arts are limited to certain object types or classes,
and does not extract undetectable information such as small part assembly, complex object manipulations and tool usage.
To this end, we make four contributions to this field.
We introduce a novel approach for extracting complete 3D assembly instructions from a video tutorial
utilizing the object’s assembly graph to infer otherwise undetectable assembly steps.
We introduce a novel visualization technique for assembly instructions combining 3D animations with video information to encompass all information from input video tutorials.
Video annotations provide information on otherwise undetectable manipulations.
We introduce a practical approach for an AR tutorial setup using commonly available hardware by utilizing a smart-phone for object tracking and a monitor as AR mirror.
We demonstrate the completeness of our extraction and visualization approach by performing a user study comparing against typical video tutorials.
Our system can be explained in three steps.
Input to our system is a video tutorial and a 3D model of the object.
Our system animates each of the parts following the assembly procedure that is demonstrated in the input video.
First, we compute the assembly graph from the 3D model.
We use the assembly graph to support the detection of the removal of assembly parts in the video.
To identify unblocked parts that can be removed, we compute blocking relations between all parts of the 3D model.
We extract the assembly sequence from the video input in the second step.
We compute the assembly sequence from the input video by analyzing the video frames in reverse order,
thus, observing the disassembly of the object and detecting the removal of object parts.
Hence, inverting the observed disassembly sequence provides us with the actual shown assembly sequence.
For each disassembly step, we use the assembly graph to identify all currently removable parts in the 3D model.
Once a complete disassembly sequence has been computed, we generate visual instructions to communicate the assembly actions.
We generate commonly used assembly visualizations such as motion lines and 3D animations by translating each part in the direction where no other parts block the removal.
Our approach is based on the registration and tracking of an existing 3D model of the assembly in the video.
We aim at a wide range of assembly tutorials. Therefore, we focus on common RGB video formats.
Initially, using GrabCut, we generate an object mask of the fully assembled object at the last frame of the input video.​
Subsequently, we derive the initial 6DOF pose from the mask, using a CNN
After we register the assembly and determine its initial 6DOF pose, our system tracks the rigid 3D assembly using an approach based on histogram-based probability maps
However, since the observed object is disassembled in a video tutorial, its 3D shape changes over time.
This will eventually lead to tracking failure as seen on the left.
To improve tracking stability, we make use of the previously determined removable parts list.
We use multiple trackers, one for each removable part, and use the tracker with the highest confidence, i.e., the one tracking a still attached part, for pose tracking.
We infer the removal of a part from the probability of its 2D projection.
Therefore, we project each removable part to image space to generate its footprint using the current assembly pose.
We compute the probability of all trackers and use the one with the highest confidence to track the visible assembly.
Then, to detect part removal, we use the current camera pose to project removable parts (illustrated in red), and compute the probability of their existence in the video frame.
This process iterates until the assembly graph is empty.
However, since we rely on computer vision, we can only monitor the presence of object parts that provide a sufficiently large, detectable footprint.
Therefore, in each frame, we first test the detectability of all removable parts by measuring the size of their 2D footprint in image space.
If a part cannot be detected, we mark it as potentially removed in the assembly graph, so that blocked, but otherwise detectable parts become removable.
Here we present a removal detection result.
As you can see in left and center images, the system tracks the scooter parts and detects removed parts by referring to the probability map.
The mask here is generated from the current parts list and the current camera poses.
Each time the tracker detects a removed part, the footprint changes and a new tracker is started.
To further improve tracking stability, we handle hands occluding the assembly by performing hand segmentation, here seen on the right.
Since multiple undetectable, small parts may block removal of a large part, we need to determine their correct removal order from video information only.
We derive this from analyzing the user's hand movements and the used tools.
We assume that the user is removing a part using a tool or with her hands directly.
This allows us to infer the removal sequence from the user’s hand and tool locations relative to the small object parts.
When presenting instructions, we reduce cognitive load by matching the appearance of the 3D model
to the one in the video by using pixel colors from the video and applying them to the model texture.
We create a texture atlas for each part, and we fill it by projecting the pixel from a video frame onto the object using the estimated camera pose.
We only project video frames in which an object has been identified.
Some parts may not be visible, in which case we additionally use an inpainting solution to estimate missing pixel colors from those within the texture atlas.
A video label is a cropped video excerpt focusing on the interaction with a part.
The label is especially helpful to communicate actions that are otherwise challenging to detect in a video
and hard to visualize in a 3D animation, such as tool usage or non-rigid object deformations.
In such cases, the video label is able to present direct visual explanations of interactions.
To crop the input video to a video label, we calculate the center and the radius of the most important area based on part, hand and tool information.
First, we detect the contours of parts, hands, and tools.
When a tool and a hand, or two hands have been detected we combine their masks and treat them as one.​
Subsequently, we compute the minimal enclosing circles on each contour.
We obtain the centers of the part and hand regions, and the radii of the circles, which enclose the detected masks.
We implemented three types of video label placement based on users’ feedback.
In the first prototype, we placed the video label on a 3D plane connected to the object via a cone to indicate a recorded camera position.
We expected that the 3D placement will allow anticipating label motions and, thus, labels would not distract users.
However, the actual user feedback was contrary to our expectations.
We revised the video label and attached it to the corner of the display, but this positioning was criticized as being too far away from the object.
In the last iteration, we placed the label center above the object to keep all of the content to the user's view.
We validated the results of our approach to automatically create step-by-step Augmented Reality instructions from an unmodified video input in a user study.
We compared the extracted Augmented Reality instructions to traditional video tutorials.
In this study, participants were asked to assemble a car model with 34 parts, attached to a wireless Inertial Measurement Unit for tracking.
After assembling half of the model using one type of tutorial, participants continued assembling the other half using the other type.
We collected these measurements and formed the following six hypotheses based on our expectations grounded in previous research.
Statistical analysis revealed confirmed three hypotheses fully and one partly.
We reject H1 and H2.
We accept H6, as all participants preferred the Augmented Reality system over the traditional video tutorial.
Not that half of explicitly stated that they liked the additional interactivity of our system.
For the Augmented Reality condition, the SUS score was clearly above the average 70, at a score of 87 or “excellent”.
Users also experienced significantly less mental effort.
While only few participants pointed out the colors matching real and virtual parts, we found indications that users actually used colors to follow instructions ,
because 37.5% participants mixed up similarly colored parts and made errors when assembling these parts.
We partly accept H5 as the temporal demand score in NASA-TLX was lower for the Augmented Reality instructions.
One fourth of participants stated that they were less pressured by the step-by-step instructions in the Augmented Reality condition, which is in line with the findings in existing literature.
Regarding the pressure for completing each assembly, we found out that participants usually tried to work at the speed of the video in the video condition,
while they took their time in the Augmented Reality condition, which may explain the lack of difference in task completion time (H1).
Overall, the study demonstrated that our system successfully extracted complete visualization for completing the task.
We put forward four design recommendations based on our findings and literature review.
First. A key to reducing mental effort is the presentation of step-by-step animated instructions. It allows users to follow instructions at their own speed.
Second. Another key is to allow users to control the viewpoint to align the viewpoint of real-world object with the viewpoint in the Augmented Reality mirror, and, thus, the instructions.
Third. For small and non-rigid objects that are hard to detect with computer vision algorithms, we can still capture the information using the original video as a label.
And Finally, using visual information directly from input videos can align the visuals of real-world object and virtual ones, which may support visual search when following instructions.
As future work, we plan to provide a web-based interface to allow content providers and consumers access to the video extraction software and the Augmented Reality mirror system.
This allows us to collect test data that follows the guidelines above to produce videos suitable for our approach for further analysis,
and will in the long term push online tutorials to the next level.​
Our approach is aimed at processing detailed assembly videos.
Currently our system can process videos that show objects that are continuously manipulated in a series of video frames without interruptions.
While cuts in the video can be handled by reinitializing the object tracking, stop-motion sequences or slideshows will require frequent reinitialization, which decreases usability.
Our system currently does not use audio information from the video.
However, a combination of extracted video annotations and voice overlay could further enhance the presented information.
The audio track could also be used to focus the automatic detection of assembly steps by recognizing audio related to certain objects or tasks.​
In the future, 3D models could also be inferred from the video itself.
Many objects that require assembly are composed of standardized pieces.
In such a case, once an object is detected in the video frame, its 3D model could be retrieved from an object database. ​
Concluding the presentation, we summarize our main points.
We automatically extracted 3D Augmented Reality tutorial from unmodified videos.
We performed a user study comparing Augmented Reality against traditional video tutorials using a complex model.
We put forward design recommendations based on the findings of our study.