Hello everyone.
Thanks for watching this video.
I am Daehwa Kim, a Master’s student at KAIST.
Today I’ll introduce OddEyeCam,
, which is a practical inside-out mobile device tracking system
to support body-centric peephole interaction.
This work is done by HCI Lab at KAIST,
in collaboration with Keunwoo Park and my advisor Geehyuk Lee.
Nowadays, mobile devices are becoming platforms for various applications.
However, their small screens make it difficult for users to use information-rich applications.
To overcome this problem, many researchers extensively studied the interaction concept
of expanding the interaction space to the vicinity of the human body.
Users could peep into a virtual workspace through a screen of mobile devices.
Also, the users could utilize their kinesthetic sense when navigating around their bodies.
Therefore, body-centric peephole interfaces have gained considerable attention.
To support such spatial interfaces, various technical approaches were proposed.
One approach was to capture a user’s face using the mobile device’s front camera.
Another vision-based approach captured motions of non-body features using a rear-camera.
Another approach utilized the motion data obtained from an accelerometer, gyro sensor, and magnetometer.
However, these approaches do not track the user’s body position.
Therefore, reliable position tracking in body-centered interfaces is difficult when the user is walking,
or the user’s head and body are misaligned.
Moreover, recent neuropsychological studies found that a human perceives the trunk-centered space
as a whole body-centered space rather than the head-centered space.
In response, we propose OddEyeCam, which tracks the body as a reference frame.
Similar to an odd-eyed cat with two eyes of different colors,
we combined two cameras with different characteristics.
Despite the depth camera’s useful function,
its field of view is too narrow to capture a peripheral scene.
A narrow view camera misses a significant portion of a body when it is close to the body.
Therefore, OddEyeCam finds the body keypoints from a wide view RGB image.
And then, partial depth information around the user’s trunk is obtained from a depth camera
to estimate the device position relative to the body.
Recently, many mobile devices include a ToF depth camera and wide view RGB camera.
OddEyeCam is expected to be a practical method to support body-centric peephole interaction for mobile devices.
Considering the limitations of earlier work, we set up four design goals.
First, OddEyeCam has to estimate the absolute location of the mobile device relative to the body.
Second, to cover various interaction scenarios,
OddEyeCam has to continuously track the device location
in a wide interaction space around the body.
Third, our system must robustly estimate the mobile device’s location without restricting the user’s behavior.
Fourth, to completely support mobile devices’ portability,
OddEyeCam must track the device position relative to the body even when the user is moving.
We implemented the OddEyeCam prototype using two cameras.
We attached a one-hundred-eighty degrees fisheye camera and an RGB-depth camera to the mobile device.
The wide-angle camera provides a whole-body image,
and the depth camera provides depth information around the user’s body.
This is the overall pipeline of OddEyeCam.
Inputs are a wide view RGB image and a narrow view depth image.
The output is the position of the mobile device relative to the body.
In the camera fusion module, we transformed a wide view image to match the narrow view depth image.
From now on, I will explain how we matched these two images.
We have two images taken from two cameras.
Our goal is to combine useful information in each image.
We applied equirectangular projection to the fisheye image to alleviate the distortion of it.
The pose estimation module finds body keypoints on this image.
While a normal RGB camera uses a pinhole camera model,
a fisheye camera uses an omnidirectional camera model.
We applied perspective projection to the wide view image.
As shown in two checkerboard images below,
we could transform the wide view image like a normal field of view image.
We placed a checkerboard sheet at 40 cm from the cameras and obtained the homography matrix
using the features on the surface taken by two cameras.
The wide view image was cropped, transformed, and rotated to match the depth image.
Finally, the body keypoints on the wide view image could be found in the depth image.
We estimated the coordinate system of the user’s body.
The origin of the body coordinate system was the center of two shoulder points.
The x-axis was estimated by fitting the 3D points on the line
from the left shoulder to the right shoulder.
To set a y-axis as a foot-to-head direction, we used the direction of gravity which is
measured using the built-in accelerometer of the mobile device.
The z-axis was obtained as the cross-product of the x-axis and y-axis.
Finally, we can track the device position relative to the body.
We quantitatively evaluated our system through three experiments.
First of all, we conducted a tracking accuracy test.
In this test, we evaluated how accurately OddEyeCam could estimate the x/y/z position
of a mobile device compared with the OptiTrack Motion capture system.
We asked participants to move the mobile device freely and evenly in the space around their bodies.
We provided visual aids on a floor and wall to help the participants to move within the
boundaries of -60 to +60 degrees.
Participants moved the mobile device when they were standing and walking.
Participants walked back and forth along the 1.5 m line on the floor.
Also, to know the effect of clothes, we asked participants to wear two different ones.
All participants wore casual apparel and a coat they brought.
The device position estimated by our system and optitrack was compared.
The four figures in the left show trajectories of the mobile device seen behind from 10 participants.
The error bar indicates the Euclidean distance between the mobile device position estimated
by OddEyeCam and OptiTrack.
The numbers in four figures indicate an average Euclidean distance.
Whether the participants walked or not did not significantly affect the tracking results of our system.
However, we observed increasing x-position errors when participants were wearing a coat.
When the camera shoots a user from the side, the thick clothes hide the shoulder, and the
camera could not capture the hidden part.
In the second experiment,
we checked whether OddEyeCam satisfied our design goals
and whether it was sufficient for various body-centric peephole interfaces.
We divided the design space into six subspaces to cover various design possibilities of OddEyeCam.
The table below shows design spaces in combinations of
the user’s walking state,
current mobile device inputs,
and OddEyeCam.
The participants used these example applications and performed simple subtasks.
From now on, I will briefly describe the applications that the participants used in this user study.
A photo app and a messenger app was in front of the participants.
Participants could drag the photo and drop it on the messenger app.
We implemented a body-centric folder, and the participants could access each folder
by placing a smartphone at each location.
A user could select an app by tilting the device.
A world map was in front of the participants.
If the participants brought the device closer to the map, the device peeped into a partial
region of the map, which is zoom-in.
For one hand tagging application, nine tagging zones surrounded the participant.
Participants could drag a notification from the closer layer
and then drop it to the tagging zone in the farther layer.
We implemented a getting direction app, which provides different information depending on
the participants’ walking state.
When the user stands still, this application provides information about the destination,
such as nearby restaurants or hotels.
Or, it gives participants how to reach the destination when they walk.
We built a two-step marking menu so that the participants can trigger body-attached shortcuts
while paying less visual attention.
After using each application in the standing and walking situations,
each participant scored statements with a 7-points Likert scale
the statements were related to our design goals.
The results showed that the participants felt continuous movement
and felt like they moved the peephole over the body-fixed contents.
Also, they could use the contents in the same location after long usage periods
and could retrieve the desired contents even in instantaneous access.
The participants could access and interact with the body-fixed contents while they walked.
The participants reported that they could easily tag the notifications
because they could remember the location of the content relative to the body even when they were walking.
The third experiment is the quantitative characterization of OddEyeCam.
To characterize our system and understand the capabilities of OddEyeCam,
we conducted peephole target acquisition tasks on OddEyeCam.
Participants could peep into the target through the device screen and then select the target
using their holding hand’s thumb.
Participants conducted the task in both standing and walking situations.
The human’s target-selection skill follows Fitts’ law.
To verify the usability of OddEyeCam, it is meaningful to confirm whether participants’
target-selection using OddEyeCam follows Fitts’ law or not.
From 12 participants’ data, OddEyeCam followed Fitts’ law for both standing and walking situations.
In other words, OddEyeCam could support a natural human-arm and hand movement
in peephole target pointing tasks.
The overall error rates were 7.7% for the standing situation and 18.8% for the walking situation.
High error rates were concentrated at the small target widths.
Now, let me summarize this presentation.
We proposed a practical inside-out mobile-device tracking system to support body-centric peephole interaction.
OddEyeCam could track the mobile device’s 3D location in an absolute manner
in both static and mobile environments with an average error of 4.3 cm.
The participants could use various body-centric peephole interaction scenarios on OddEyeCam,
and these scenarios required continuous tracking in a large 3D space around the user’s body.
The peephole target selection task on OddEyeCam followed Fitt’s law, and this suggests that
OddEyeCam could support a natural human-hand movement.
This is the end of my presentation.
Please see our paper for more details.
Also, please feel free to contact the authors through the below information.
Thank you for listening.