Hi, I am Xinzhe Yu.
I hope everyone is doing well all things considered.
I am a postgraduate student from Beijing Institute of Technology.
And this talk is about scene-aware behavior synthesis for virtual pets in mixed reality.
This is joint work with Wei Liang, Rawan Alghofaili, Yining Lang and Lap-Fai Yu.
So let's get started.
In this paper, we propose a novel approach to synthesize virtual pet behaviors by considering
scene information, which enables the virtual pet to behave naturally and rationally in
mixed reality.
Here is an overview of our our approach.
The framework includes two components: data-driven pet behavior generator and pet behavior instantiation.
The input is a real scene and the output shows some examples of the generated pet behaviors
with each color representing one behavior type.
Next, we will introduce the details of our approach.
Phase 1: data-driven pet behavior generator.
First of all, according to the defined pet behavior categories, we use a cat indoor (location)
dataset and annotate the dataset with behaviors.
After that, we train a LSTM network to learn the pet behavior patterns from the dataset.
After the learning process, we generate the behavior sequence using the LSTM network.
Phase 2: pet behavior instantiation.
To achieve the geometry and the semantics of the scene: we perform two steps.
First, our approach scan the room via Microsoft HoloLens, so that we obtain a 3d model of
the room for scene geometry.
Second, we detect all related objects for scene semantics.
Then, according to the obtained the scene semantics, we generate a physical location
for each behavior to take place in the real scene.
Last, we instantiate paths for location switching, enabling the virtual pets to move around in
the real scene.
Here we show some results.
Living room.
In this clip, at first the cat was idling by the window.
Then it walked to the table to perform the same behavior.
After that, the cat went to the scratch post for scratching and then back to the window
for idling.
In this clip, the cat was eating at its pet bowl.
Then it went to the table to perform an idling behavior.
After that, the cat jumped onto the couch for a short time idling.
Finally it walked to a litter box for soiling.
Bedroom.
In this clip, at first the cat was idling by the window.
And then it jumped onto the bed for a resting behavior.
In this clip, the resting behavior occurred both on the bed and on the cushion.
Kitchen.
In this clip, the cat was first scratching on the scratch post.
After a while, it transferred to the counter for an idling behavior.
In this clip, the cat transferred its behavior from eating on the bowl to a long time idling
on another location.
Now it was performing an idling behavior on the table for a while.
After that, it transferred to the counter for the same behavior.
Evaluation.
We conducted user studies to evaluate various components of our approach, consisting of
behavior generation, location instantiation, and path instantiation.
We show the detailed results in our main paper.
That's all.
Please refer to our paper for more details.
Thanks for watching.