hello all thank you for tuning in to our presentation on Stargazer an interactive camera robot for
capturing how-to videos based on subtle instructor cues. People often watch how-to videos to learn
useful physical skills for workplace and for home. However making a good how-to video for teaching
these skills is not straightforward. To show why, let me first play a well-made tutorial video
Then prepare your leaks just take your knife and go down through the center turn it over
and again into quarters to get all that opening up and then. You may notice how this video clip
used various combinations of camera positions and angles to clearly show both the details
and the overall action of the chef. However this high quality does not come for free
it is based on complex equipment and a whole team of camera operators
in this picture you see three cameras and two camera operators who can dynamically adjust
the cameras to follow what is going on in the kitchen from different angles and zoom levels
this kind of resources is not available to most physical scale tutorial video creators
in contrast they often have to go through effort for camera setup processes let me play
a clip to show how much setup this instructor needs to do just to film a very brief action
so the goal of the Stargazer project is to help physical skill instructors
make high quality informative tutorial videos without a dedicated camera crew
we use the highly articulated robotic arm that can automatically track and capture regions of
interest such as the instructor's hands. This is the footage recorded by the robot.
Recognizing that the instructor knows what is the relevant visual content to be presented
to the Learners we want the instructors to be able to control what the robot tracks while
explaining and demonstrating the skills. However we do not want them to perform any kind of input
action that is only for controlling the robot but do not add to the content of the tutorial
As this will take away the instructor's time and cognitive resources for teaching
Our key idea here is to watch for subtle cues from the instructors
what makes these cues interesting is that they can be seen as part of the tutorial content itself
let me show you two concrete examples from the tutorial videos our participants created
use the medium-sized wrench shown here you can look closer it is hexagon hexagon shape which is
the same... You can see how he was able to naturally use speech to have the camera zoom in onto the
tool in his hand. In a second example pay attention to how the camera transitioned from looking at
the material on the table to her hands. ...and the orange one I'm gonna use it for a mouth for a duck
and these are for the eyes
let's start from ...
so how did we achieve these interactions we started with understanding what kind of
behaviors instructors may want from a camera robot and what aspects of these behaviors they may want
to control. We studied 50 physical skill tutorial videos shot by shot focusing on the subjects that
is what is captured by the camera the camera angle whether the camera is looking straight
at the subject or from a higher or lower position and zoom level that is how much area the subjects
takes up on the camera frame. Our analysis revealed some common patterns physical skill tutorial
videos tend to focus on small set of subjects first the hands of the instructor manipulating
objects. These shots communicate most of the actions to complete the physical task
another popular kind of shots show the instructor themselves talking to the camera in the middle
you can see these two examples. These shots try to simulate a face-to-face conversation finally the
other common type of subjects highlight objects and sometimes particular details on objects
We also find that they use two common camera angles include the standard angle meaning
that the camera is approximately level with the subject and high angles that is looking
from a higher position at a subject. Finally we notice that the instructor use at least
two Zoom levels to show overviews and details respectively Now we move to the cues that can
be used to trigger changes in camera parameters implicit cues include head and hand positions
which naturally come from the need to capture the instructor's hands and their faces in the videos
we also included torso orientation to increase the visibility of the instructor's actions we further
looked at cues that instructors commonly used to direct the attention of their students to various
aspects of a tutorial gestures are very common kind of communication signals for this purpose
following these conventions we use gestural cues to signal to the robot to change subjects
for example a pointing gesture will prompt the robot to focus on the object being pointed at
people also commonly use language instruction to direct others attention in this picture the
plottering structure can verbally suggest a boy to pay attention to how she shapes the top of the
piece following these behaviors we listen to instructed speech and look for sentences that
explicitly suggest changing camera zoom levels or angles to set the robot's camera parameters
our camera robot supports three type of shots as informed by the video survey
the first is the instructor shot showing the instructor talking to the camera then
the action shot showing the actions to complete the task and object shots to highlight an object
let me walk here through an example to show how the robot change its Behavior based on its
structure input here our instructor is giving a tutorial on how to make a 3D printed lamp
hello today I'm going to show you how to do that
take
most of the time the robot tracks the instructor's hands to film
The Action he is performing as you can see our top cover is very thin
now the instructor moves to the base of the lamp
this is our base of the lab the camera tries to face perpendicularly to the Torso of the
instructor the base is thicker than the top cover because we don't want
light to go through the veins the instructor wants to shove this socket to the audience
this is our socket he can risk his hand to transition to an instructor shot
which focuses on his face and the object he is presenting we bought it from the hardware store
at this point the instructor wants to highlight the learner a light bulb he's going to use
let's take a look at this light bulb
he can point to the light bulb to have the robot capture an object shot
let's take a closer look he can further use speech to make the robot increase
the zoom level the light bulb has yellow unit the instructor can also
use speech to suggest changing camera angle and now let's look from the top
the camera responds by transitioning from a standard angle shot to a high angle shot
let's start installing the ball
for implementation we used standard connect rgbd Post tracking to locate the regions of
Interest such as hands and hand positions they we used a vision based hand tracker plus a custom
machine learning model to find pointing gestures finally We performed field shot classification
with a large language model to identify sentences that suggest camera angle and zoom level changes
we solve camera path planning as an optimization problem at a high level our method calculates how
much a possible next camera position violates a set of criteria with a cost value J the optimal
next camera position XT Plus One will be the one that violates these criteria as little as possible
and thus has the smallest J we perform the user study with 16 structures each of them made a four
to seven minutes tutorial video a distinct topic using stargazer with thumb training all of them
were able to make the videos with one or two ticks please see our supplemental materials which is
available on SM digital library and the project website for videos produced by our participants
I think there is much interesting work lying ahead of this line of research
star Cruiser currently looks at a carefully selected set of instructor signals
we're interested in expanding the range of signals that our robot can understand especially those
other implicit in speech or actions but can be inferred with some level of social intelligence
right now is a particularly good time to do this kind of research because of the recent progress
in large language and vision language models we're also interested in going Beyond videos
work at this year's Kai already looks at capturing objects from multiple angles in video calls
the mobility and precision of robots can potentially makes it easier and possibly
facilities 3D captures of objects and actions using techniques such as neural Radiance fields
finally we would also like to explore fitting robot cinematography bettering to content
creators existing workflows for example adapting to individual Creator styles with that I would
like to conclude my presentation thank you for your attention please see your paper and our
project website for more details a QR code for the project website on the right side of this slide