- Hi everyone, thank you for the introductions.
My name is Mose Sakashita.
I'm a undergrad student in the University of Tsukuba
and I am going to talk today about telepresence
user interface for puppetry that allows you to experience
you as a puppet.
This work was done with my professor Yoichi Ochiai
and other four students, Tatsuya, Amy, Ippei, and Keisuke.
First, I'm gonna show you a little bit
what the system can do.
This is a telepresence interface for performing
of puppetry.
The telepresence system transmits the performer's body
and facial movements to a puppet
and provides visual and audible feedback to the performer.
Puppets and characters are used in various contexts,
such as TV shows, attractions in theme parks,
communication tools, and education tools.
Puppets expressing thoughts and feelings have
always attracted people and,
and to challenge their imagination.
In the long history of puppetry, several methods
for manipulating a puppet have been developed.
Some of the simplest ways to perform puppetry include
wearing a puppet directly on the hands,
and connecting strings between a puppet and the performer.
But in most cases, performers manipulate the puppet
with their hands above a table, where the rest
of the body is below the table.
This poses a challenge in coordinating the actions.
It is also hard to keep puppet's gaze in the right direction
because you can only imagine what the puppet sees.
So, puppetry requires considerable training
to manipulate a puppet.
Another way to show puppets' presence
is to wear a character costume covering their whole body.
Please don't tell your children the secret.
Anyway, in this case, all you have to do for performance
is to move your own body.
Moreover, actors can look at what the character sees
with their own eyes and then they can have face to face
interactions with the audience.
It's quite easy to perform and play a role of it.
However, all of these conventional methods
have physical limitations.
For example, puppeteers needed to perform
exactly where the puppet is situated.
Actors are required to be inside a costume,
to change expressions and motions,
or they must be nearby a puppet or audience.
It is also difficult for actors to manipulate both
very tiny and very huge sized characters
and unique shaped characters as well.
Researchers proposed several input systems
for animating a puppet.
For example, using a webcam, tongue joystick device,
and with motion.
Computational puppets are also designed basically for
social communication.
They facilitate the communication at a distance
among children and family using a puppet.
But the main purpose of these puppets is to communicate
with others, at a limit,
not to show performers to audience,
and they don't focus on providing immerse,
immersive experience that much.
The concept of our human to non human immersive
telepresence affords new experiences to performer
and audience.
Combining puppet shows and telepresence technology
can remove the physical limitations
and puppets can be manipulated at the place
where the performers want to be.
Based around the idea, we proposed
our telepresence system for puppetry.
Performers are able to manipulate their puppet
using their body, see the images from the puppet's eyes,
and have a visual understanding of the puppet's ambient.
This immersive experience allows them to act as if
they became a puppet and enjoy performing.
Moreover, beginners who are not trained to manipulate
puppets can also perform a puppet show in an easy way
using the system.
I'm gonna talk a little bit about the implementation
of the system.
There are mainly two parts of the implementation.
One is sensing and transmission,
and the other is audiovisual feedback part.
First, I'm gonna talk about the sensing and transmission.
The robot's neck is controlled according
to the performer's facial movements.
We use head tracking sensors in the HMD
to acquire the head rotation.
The angles of rotation are sent to the servo motors
in the robot's neck.
In addition, we also use the Kinect depth camera
to capture the movements of the user's arms.
We applied a low pass filter to stabilize
low resolution and noisy input data from the Kinect sensor.
The puppet synchronizes with the movements
of the performer's arms.
We implemented an array of 10 photoreflectors
to detect the open or closed state of the performer's mouth.
The array is used for detecting the lower lip's position.
Based on the position our full software controls
the puppet's mouth.
Inside the mask there's also a microphone to capture
the performer's voice.
This gives the impression that the puppet
is actually speaking to the audience.
This figure is the visualized values of the sensors.
You can see that the servo motors are synchronized
with the performer's mouth.
Next I'm gonna talk about the audiovisual feedback.
The robot carries two web cameras in place of its eyes.
These cameras stream live video to the head mounted
display worn by the performer.
So, performers are able to see the images
from the puppet's eyes with their own eyes
to understand visual,
understanding of the puppet's environment.
All the components of the puppet's bones
are three D printed, meaning that the robot
can be designed at a relatively low cost.
The puppet robot has two DOF arms, one DOF mouth,
and two DOF neck.
So, the entire body of the puppet is a seven DOF robot.
I'm gonna talk about the user study we conducted
to understand how the system affects the performer
and audience.
We needed to investigate the effectiveness of the system
in terms of both the performer's side and the audience side.
For performers it is crucial that the system
is usable and enjoyable and not extremely exhausting
and from the perspective of the audience
it is also important that the puppetry using
the system is fun and lifelike.
At first, we investigated how performers felt
when they performed puppetry using our system.
We prepared a simple stage and asked participants
to manipulate a puppet freely for a few minutes,
and then we asked the participants to answer
some questions about the usefulness and enjoyment.
We compared the results between the beginners and experts
who are familiar with hand manipulation.
I wanna show some of the results of the user studies.
The first question pertained to operating a puppet
from the puppet's point of view.
Both participants rated it positively on an average.
According to the participants' descriptions
there were positive opinions to benefits of it.
A puppet, a professional puppeteer says that
she usually tried to imagine the puppet's point of view
which is so hard, but with the system she could
see everything from the puppet's point, viewpoint
and that was useful.
A beginner says that he felt as if he became the character
which is a similar feeling that
when he wear a costume.
The second question was about moving their body
for manipulation.
The statistical tests show that the difference was
significant between general and professional participants.
This result indicates that beginners who are not familiar
with hand manipulation,
manipulation feel easier to manipulate the puppet
by the system, unlike professional puppeteers
who are accustomed to the conventional method.
One professional puppeteer says that she could feel
experience as if she became the puppet
and it was easier than by hand.
A beginner mentioned that in a conventional method
he didn't have a technique to move its hands up and down.
The third question was about the overall usefulness
of the system.
There was also a significant difference between
professional puppeteers and beginners.
Overall, the professional puppeteers rated the system
neutrally and we think that this is because they are
already good at manipulating a puppet by hand,
but aren't familiar with the system.
A beginner mentioned that with the system
she could express her feelings.
According to the comment, she felt like she became a puppet
rather than manipulating it consciously.
This question was about the enjoyment of the system.
There was also no significant difference
between professional puppeteers and beginners.
In both professional and general participants
the averages were over six points.
This result shows that all participants
felt enjoyment using the system.
Participants commented on their enjoyment of the system
from different perspectives.
One of professional participants enjoyed the experience
of having the puppet's mouth synchronized
with the user's mouth, and the other enjoyed
seeing that the audience came close to the puppet.
Secondary, we investigated how the audience felt
when they saw the puppetry played with our system.
We recorded videos of the hand based
and telepresence based puppetry,
acted by professional puppeteers,
and then asked participants to watch and compare
these puppetries.
We asked questions about the enjoyment, cuteness,
and lifelikeness of each puppetry.
The average scores of hand manipulation were higher
than those of the system.
But the statistical tests for each question showed
that no significant difference existed
between the system based and hand based puppetries.
We think that the mechanical movements of the puppet
and the sound of servo motors affected
the results negatively.
Next I wanna talk about several limitations
of the current puppetry system.
In conventional puppetry a puppet can move around
on the table freely, but the puppets manipulated
by our system are fixed on the stage
and they cannot move from one end to another.
Professional participants mentioned that
they would become troubled when they used
some props to tell a complicated story.
To address this, we can use the center of gravity
detection system like Nintendo Wii Fit
or simply other orientation detected by Kinect
can be used for the movements on the table.
Professional puppeteers really care about
the appearance of the puppet.
So, it is useful for them to see how a puppet
looks using our system.
But the system allows only the,
shows only the images from a first person viewpoint
and does not have function to provide feedback
for the state of its appearance.
We think it is possible to design software
to calculate and display the state of the puppet's body.
We could also address this issue by showing images
from a fixed point camera in the head mounted display
that capture the puppet objectively.
Some participants also commented that the puppet
looked mechanical or artificial somewhat
because of the way it moves
and the sound of the servo motors.
The puppet is a seven DOF robot
and this is not sufficient to express
subtle human emotions.
The system could address this issue
by adding extra servo motors, and regarding the sound
of motors, playing some background music
may mitigate the sound to some extent.
Most participants feel some fatigue when they
used the system.
This is mainly because of VR sickness
due to the latency of video displayed in the HMD
and its weight.
Oh, sorry.
In future applications of telepresence system
omnidirectional cameras can be used to display video
remotely because the video can be cropped and displayed
to users without being affected by the servo motors.
But it might be also problematic because the viewpoint
becomes slightly different from what a puppet actually sees.
Hand based puppetry can be performed at a low cost
in a small space, but the proposed system
requires approx 1.5 square meters for the puppet,
performer side.
This is because the system contains a depth camera,
laptops, and other components.
We could use algorithm for estimating human pose
from RGB images that doesn't need wide space.
The cost of the system is also relatively high
because of these equipments.
For the general public the HMD can be replaced
with a low cost version such as Google Cardboard.
Finally I'm gonna talk a little bit about future work.
Recently in theme parks interactive attractions
with three D animated character are common
but they are rarely interactive attractions
using physical puppets.
The attractions using of technology could encourage
repeat customers by changing the content each time
they ride on the attraction.
Our immersive telepresence system would also be useful
and enjoyable for attending a conference,
or communicate with family at a distance.
We believe that the physical puppet allows
people opportunities to express their thoughts
and convey a number of information to others.
So, for the bones of our puppet are manually designed
and adjusted to the size of puppet.
In future work it is very interesting if the bones
inside a puppet were calculated computationally
giving several parameters like its size
and number of joints.
In summary, we implemented an immersive telepresence
system for puppetry, and conducted several user studies
to investigate how the system affects the performer
and audience.
The results show that the system facilitates
the manipulation of puppets especially for beginners
and the system allows all the performers to enjoy
performing puppetry.
Finally, we conclude by discussing the limitations
of the system and possibilities for future work
beyond puppetry.
This is the list of research members of this project.
I really thank all the members who worked together.
We will give our demonstration this evening,
so please do come and experience you as a puppet
in the demonstration.
Any questions are welcome.
Thank you.
(audience applauding)
- [Audience Member] Hi, Jeremy Werner from UC Berkeley.
So thank you for presenting this work.
Often puppeteers may control multiple puppets
at one time but this I think just one puppet
with one puppeteer.
Do you think that this would be possible to extend
to maybe controlling two puppets simultaneously or
do you think that's infeasible?
- Yeah, yeah, I think this is maybe,
this is one of our
possibilities, application, because the current system
is sending the data to only one puppet
but it can be expanded to
two puppets or three puppets at the same time.
I think it is possible and it will
expand the entertainment and yes,
I think so, yes.
- [Audience Member] Hi, (mumbling) Bordeaux, France.
So, I really love it.
I have a question about hybrid approaches.
Do you think about keeping the helmets while
being still under a table or for example I don't know
having a, no, mostly that.
Do you think about that, having the helmet
but stay in the same position like that?
Do you think it makes sense just for the feedback
without the activation, let's say?
Do you, do you understand?
Maybe I'm--
- Sorry, I, I--
- [Audience Member] It's okay, I will ask
you at the demo reception, sorry.
- [Man] He's saying just to have the helmet
but you're under the table, is that a good scenario?
- Why, why the, the--
- [Audience Member] It's okay, I will ask him later.
- Sorry.