Hello everyone. My name is Liu Jingyuan. I will present our work PoseTween, which is a system for augmenting visual effects to human action videos for novice users.
There are existing professional software for adding visual effects, such as adobe after effects. But they would require experienced users with trained skills.
Some light-weight applications support adding virtual elements that track the movements of subjects.
But such tracking-only effects cannot handle more complex interactions between subjects and virtual elements.
In this work, we aim at studying how to easily add virtual elements that interact with a moving subject in a video.
One goal is to precisely control the movements of virtual elements, to achieve more natural interactions with the subject than only tracking.
On the other hand, since the human body is flexible to depict complex motion paths,
we can also use the human movement to roughly guide the movement of the virtual elements to simplify the visual effects editing process.
To achieve these goals, we model the visual effects as tween animations of the virtual elements, and utilize the human poses in the video to guide the tween animations.
We first demonstrate how PoseTween works with this dribbling basketball example. The input to our system is a source human action video.
The pose of the subject at each frame can be obtained by existing 2D pose estimation methods, like OpenPose in our implementation.
Given the source video, a user edits the virtual elements at keyframes with respect to the human action to create an animation of the virtual basketball.
Then in the augmented video, the resulting animation of the virtual basketball is jointly determined by the human action and the user-specified animation.
In the following, we explain the main idea of pose-driven tween animation.
In this example, the basketball is driven by the right wrist of the subject. At each frame, the right wrist determines a local frame for the basketball.
The origin of the local frame is rooted at the position of the right wrist.
At a few keyframes of this action, a user specifies the positions of the basketball by dragging it to the desirable positions on top of the video.
The positional difference between the basketball and the driving joint is represented as the position displacement within the local frame.
Then at intermediate frames between the two keyframes, the position of the basketball is a combination of the position of the local frame,
and the interpolated position displacement within the local frame. In this way, the use of the joint position reduces the need to model complex basketball trajectory,
while the position displacements in the local frame facilitate the relative motion between the basketball and its driving joint.
Besides position, we also allow the orientation of a virtual element driven by the orientation of a limb, like this racket driven by the right forearm.
The angle displacement of the racket in the local frame allows it to be not strictly aligned with the forearm to avoid rigid-looking effects.
We provide a user interface for editing and previewing such pose driven animations. As the user specifies the keyframe positions and orientations of the elements,
the driving joint and driving limb are inferred automatically as initial guesses to reduce user interventions.
Here are a few more examples made with PoseTween by the participants in our user study.
Besides adding visual effects, PoseTween can also be used for other applications, such as storytelling and making tutorial videos.
Our pose-driven animation is based on the computed human poses from the videos.
However, the existing 2D pose estimation methods might fail under certain circumstances.
As shown in the figure on the left, the joints might be missing or inaccurate due to occlusions.
We thus also provide a tool that allows users to manually correct joint position errors.
Here some joint positions are inaccurate due to the occlusion. When a user manually corrects the joint positions at one frame,
the changes will be propagated to the neighboring frames to reduce user interventions.
Please refer to our paper for more details of methods and more results. Thank you!