Hello, I’m Beomsu Lim, a Master student from Interaction Laboratory at POSTECH, South
Korea. My research topic is “Image-Based Texture
Styling for Motion Effect Rendering”.
A motion platform provides the vestibular stimuli that elicit the sensations of self-motion
and thereby improves the immersiveness. The motion platform technology is now widely
used for various entertainment purposes including amusement rides and 4D films.
A representative example is a 4D Ride, which presents a video of POV shots and motion effects
synchronized with the camera motion in the video.
Previous research efforts resulted in a few automatic motion effect synthesis algorithms
for POV shots. For example, Lee et al. termed motion effects
for POV shots as “fast-camera motion effects”, and developed an automatic synthesis algorithm
for them.
Although effective in generating gross motion effects, motion effect synthesis algorithms
for POV shots do not consider fine features on the ground, such as a rough or bumpy road
In contrast, manually designed motion effects express such details by the high-frequency
add-in, eliciting sensations of roughness or bumpiness.
The absence of such a high-frequency textural signal can degrade the immersiveness and realism
of the applications.
In this paper, we propose an algorithm for styling the gross motion effects using a texture
image. Our algorithm transforms a texture image into
a high-frequency textural motion and uses the textural signal to stylize a fast-camera
motion and add details.
Our algorithm takes a texture image, a camera speed, and a pre-generated fast camera motion
effect, and then generates a stylized motion command through three steps: style motion
effect synthesis. , compensation of style motion effect, and
merging fast camera motion effect and style motion effect.
In the first step, a style motion effect is synthesized from the 2D texture image input
and the camera speed estimated from the video.
To generate a textural signal that preserves the spectral content of the input image, we
apply Linear Predictive Coding (LPC) to the individual rows and columns of the image and
obtain a large number of predictive filters. A texture model is generated by using the
average predictive filter, and we synthesize the textural signals using the averaged prediction
filters. We map the textural signals in the horizontal
and vertical direction, to the motion commands in the roll and heave channels, respectively.
In real situations, a vehicle running on the road will shake more frequently and violently
if its speed increases. This is reflected into the final style motion
by resampling the texture signal in the time domain and scaling it as to the camera motion
velocity. After scaling and resampling the textural
signals, we can get the style motion from the 2D texture image.
The second step aims to compensate for the effect of the fast-camera motion on the perceived
magnitude of the style motion. In this step, the perceived magnitude of the
style motion affected by the fast camera motion is equalized to the perceived magnitude of
the compensated style motion when it is presented alone.
We performed a perceptual experiment to obtain the perceived magnitude function of high-frequency
motion stimuli when mixed with low-frequency motion stimuli.
Then, we adjust the amplitude of the style motion effects in response to the amplitude
of low-frequency fast camera motion.
In the third step, the style motion effect is merged with the fast camera motion effect
by Model Predictive Control. As you can see in the upper plot, the simple
addition of style motion and fast camera motion can exceed the limits of the motion platform
MPC can combine two motion effects such that the merged motion fits into the motion platform’s
workspace. We also integrate the vestibular perception
models into MPC so that it preserves the perceived angular velocity and the perceived linear
acceleration.
To evaluate the effectiveness of our texture styling algorithm, we conducted a summative
user study. Participants experienced combinations of audiovisual
content and motion effects. One was commercial effects manually designed
by experts. Other one was motion effects generated by
the fast camera algorithm by Lee et al. The other two were our texture-styled fast
camera algorithm with or without the compensation, which was the second step of our algorithm.
Then, participants rated six subjective metrics: enjoyment, immersiveness, realism, fatigue,
harmony, and preference.
We used three commercial 4D ride films purchased from CJ 4DPLEX.
These videos were chosen for the diversity of the floor surfaces in the videos.
All of the three films were cut into 20 seconds clips, where the ground texture was well seen.
The texture images for each video were also selected differently to fit with the video
content.
The two motion effects of our algorithms showed better performance than the fast camera motion
for most metrics. This result supports that our approach is
successful for enhancing viewer experiences. However, the motion effects of our algorithm
could not outperform the manually designed one.
In CHINA, the two textured automatic effects showed better scores than the fast camera
motion effect, but with no significant differences. It can be due to this video CHINA accompanying
the most dynamic and strong motion effects than the other videos, as commented by 5 participants.
This could have made the viewer experiences enhanced by the addition of the textural motion
effects less contrasting.
In RAIL, our algorithm significantly outperformed the fast camera motion for most metrics.
This evidence supports the benefit of the textural styling of motion effects.
According to the participants interviews, the lacking in bumpiness in the fast camera
motion caused the sense of speed to decrease. Also, two of participants said that the up-down
shake of our algorithm’s result elicited more immersive experiences.
However, the continuous and stronger shake in our algorithm could feel unrealistic and
caused more fatigue, also as commented by there people.
In RACE, None of the motion effects had a significant difference.
It can be because RACE used the smoothest texture image.
Interestingly, 7 people reported that the bumpiness of manually designed motion was
inadequate\hfor the video since the racing car ran on the clean asphalt road.
The proposed algorithm has several advantages. First, texture styling can be done easily
with only a single texture image. Second, a perception model for high-frequency
motion is explored when it is presented with gross low-frequency motion.
Third, MPC used for merging low- and high-frequency motion effects well respect the motion chair’s
working range while preserving the perceived angular velocity and linear acceleration.
Finally, the effectiveness of our algorithm has been verified by users.
Thank you for watching, and please check out our paper for more details.