Hello everyone, today I’m going to talk about our work MoiréBoard, a stable, accurate
and low-cost camera tracking method.
Commercial VR headset, use dedicated hardwares, like the laser base stations, to track you
head movement. It cost thousands of dollars so that you can
have a high quality immersive VR experience.
On the other hand, low-cost, mobile-powered VR devices has attracted a lot of attention.
However, the tracking accuracy is not comparable to the commercial VR systems.
The main reason is that, for mobile VR, you can only use your camera, plus some built-in
sensors to estimate your device location, but commercial VR systems use customized hardwares.
To fully unleash the power of mobile VR, we present a low-cost yet accurate camera tracking
method, that enables high accuracy device tracking, using only a single camera.
When we talk about camera tracking, it is an essential problem in computer vision, the
goal of camera tracking is to estimate the camera position from a photo taken by the
camera.
Traditional camera tracking is based on the principle of feature points detection.
When we take photo at different angles, we will get different images.
And if we have a way to detect the feature points, like the corners of the cube in this
example We can estimate the camera position by using
the 2d to 3d correspondences.
The problem of feature point based tracking is that, they are not very accurate when camera
movement is small.
Let’s consider this example, these two images are taken by different cameras, but the camera
location is very similar. If we detect some feature points, like the
corner of the cube.
We can see that the pixel coordinate of this corner is close. Since the detection always
have some errors, when the camera has very small motion like this, the signal-to-noise
ratio will be small and the recovered motion will have large errors.
So we are thinking about how to amplify this signal. How to make the images have strong
changes when camera moves only slightly.
I’ve been thinking about this problem for a long time, until one day, I was sitting
in front of my window. When I was looking at my window, I found this
kind of mesh structure. And when I’m placing another mesh structure
in front of my window screen, and seeing through this two meshes, I can see some interesting
patterns.
The most interesting thing is, it has the amplification effect.
When I move the camera slightly, the pattern changes largely.
So I start to wonder that,
Can we infer the camera position from the window pattern? For example, here we have
two consecutive frames captured by the camera. you know that the camera position is very
close, based on the reference objects outside the window, but the window pattern here ,as
you can see, is very different.
In fact, this is an interesting optical phenomenon called moiré effect.
The moire effect is caused by the superposition of two repetitive patterns.
Consider two square grid pattern here, grid A and grid B.
When they overlap with each other, we can see moiré pattern appears.
When we fix one grid and allow the other grid to move, moiré pattern will also move.
And the moving direction of moiré pattern is depends on the moving direction of the
grid.
Let's consider another scenario, when a camera see through two grid patterns. The captured
image will produce moire effect.
When the camera moves along the right direction for a certain amount.
The moire pattern of captured image will also change, due to the projected grid have different
moving speed on the image.
We have developed a full theory of moire pattern under camera projection. Based on our theory,
we found that, if we can measure the grid spacing and grid period for both grid, in
physical space. And measure the moire period, and moire peak displacement, in image space,
measured by number of pixels, we can use those measured amount to solve the camera position.
This computation assumes the camera is perpendicular to the grid plane. For this kind of frontal
view camera, the moiré pattern will always look like a square grid pattern.
But when the camera is tilted, because of perspective distortion, the captured image
may look like this. so the moire pattern is no longer a square pattern.
Thus, we cannot measure the period of moire and other quantities to recover the camera
location.
Fortunately, if we can detect the four corners of the moire image, and apply perspective
transformation on it and obtain a rectified image. We can theoretically prove that, the
rectified image will be identical to the image captured by a frontal view camera at the same
location. This allows us to recover the camera location regardless of its viewing angle.
In summary, our method has the following benefits. We provide high tracking accuracy due to the
moiré amplification effect. And our method is calibration-free, we do
not require any prior knowledge of camera intrinsics.
Last, our method is very fast and easy to implement, the camera position can be computed
directly from our derived formula. For detailed derivation, please refer to our paper.
we also built a real prototype to validate our idea.
We use a LED display which shows a grid pattern as the first layer , and the second layer
is a 3D-printed mesh structure. Here the LED display can be replaced by a
printed paper for further reduce the cost.
When the camera sees through the mesh, we can see the moire pattern clearly.
The four visual markers here is to help us identify the region of moire pattern.
Once we identify the region, we apply perspective transformation to convert it to a rectified
image, some attributes like the period of moiré pattern and moiré peaks displacement
can be measure on this rectified view.
As we can see here, the moiré pattern changes quickly and smoothly as camera moves.
Last, we show a real mobile VR application. We insert the phone into a mobile VR headset.
Then, we use the view captured by the camera, to reconstruct the camera motion. And use
the reconstructed camera motion to render a virtual scene that can be displayed on the
phone.
As you can see here, the motion of the virtual scene, matches the motion of the camera view
very well, even at some extreme angle.
Although here we motivated by the mobile vr application, our method is a general camera
tracking method which can be also used for other applications like robot navigation.
This is the end of my talk, thank you for your attention.