Greetings, travelers! My name is Ruofei Du. I am a research scientist at Google. Today,
I'm going to present our open source magic: DepthLab, real-time 3D interaction with depth maps
for mobile augmented reality. Augmented reality has gained mainstream popularity
on mobile devices with thousands of AR apps such as Pokemon Go, Snapchat, and IKEA Place.
These apps are typically supported by Google's ARCore or Apple's ARKit to place virtual objects
anchored on flat physical surfaces, or to invoke experiments as a reaction to detected AR markers.
However, is direct placement and rendering of 3D objects sufficient for realistic AR experiences?
Not always! For instance, without awareness of occlusion,
virtual content looks like it's pasted on screen rather than in the real world.
With depth map and occlusion-aware rendering, the virtual cat in augmented reality looks much more
realistic than the formal one. Here, correct occlusion helps ground content in reality
and makes virtual objects for you as if they are actually in your space.
Next, we wonder: how can we bring these advanced features to mobile AR experiences.
With Google's ARCore Depth API available on millions of android devices,
we wonder: is there more to realism than occlusion? What about surface interaction,
realistic physics, and path planning? We are used to these concepts in typical 3D games,
why not in AR on our phones as well? Next, please enjoy (a short video). DepthLab
encapsulates a variety of depth-based UI or UX paradigms including geometry-aware rendering,
real-time visual effects such as rain, snow, flooding, and re-lighting, and surface
interaction behaviors such as surface splatting, real-time mesh generation, and physics simulation.
DepthLab is available on a large number of Android devices with a single RGB camera.
We compute live depth map by the depth from motion algorithm introduced in SIGGRAPH Asia 2018.
Instead of learning depth from a stereo pair of camera, we use
depth from motion providing stereo images across time as the phone moves through the environment.
However, this presents additional challenges compared with traditional stereo. While
traditional stereo has full control of how the cameras are arranged with a wide camera baseline.
Stereo from motion cannot control how the user moves the camera,
also cannot guarantee FoV (field of view) overlap, velocity match, motion blur, auto focus and etc.
Users likely to move very little. They want to arbitrarily move the mobile phone.
So the important considerations include: which keyframe to choose and how to maximize
information generated from each frame. Keyframes are chosen from a pool to give the best keyframe
to give the best possible depth for the current frame oftentimes the keyframe is just a few
centimeters away from the current frame then we use stereo matching algorithm providing a sparse
and noisy depth representation that needs to be filtered interpolated and smooth we call it raw
depth in the upcoming AR core the depth image is matched for every camera image professed by
AR core at 30 hours our application can acquire the latest depth image for the current frame for
more details please check out our SIGGRAPH Asia 2018 paper - depth from motion for smartphone AR.
Currently, our depth-16 image format allows for depth sensing up to 8 meters and the best depth
measurements are between 0.5 meters to 5 meters. However, even with real-time depth map,
there's still a large gap between the raw depth data and the typical expertise of mobile
application developers who are not experienced in handling depth data. So in DepthLab,
we process the draw depth map from ARCore API and provide customizable and self-contained
components for mobile AR developers to build more photorealistic and interactive AR applications.
To explore the design space, we conducted three brainstorming sessions with a total
of 18 participants and proposed a total of 39 aggregated ideas. These participants include
researchers engineers and us designers who have worked on AR or VR related projects.
We listed these ideas in supplementary material. Please feel free to scan the QR code.
We'll also show the QR code in the end. When developing DepthLab, we architect
and implement a set of data structures and real-time algorithms for mobile AR developers.
We generated three kinds of data structures: depth array, depth mesh, and depth texture.
Depth array stores depth in a 2D array of 16 bit integers on the CPU, which
which is typically 160 by 120 pixels and above. Second, depth mesh is a real-time triangulated
mesh generated directly from the depth map. We'll introduce later how we generated the depth mesh.
Third, the (depth) texture, which is basically a GPU texture decoded from the depth array and
interpolated to larger resolution. Based on these structures, we classify our
DepthLab components into three categories: localized depth, surface depth, and dense depth.
Localized depth uses the data array to operate on a small number of points directly on the CPU. For
example, by converting between the screen space with the word space space, DepthLab provides
a 3D oriented cursor. The cursor orients according to normal vector of the physical surface
and details about its distance to the ground and to the camera. Computing
useful normal maps of low-resolution and the coarse depth maps can be very challenging.
A simple cross product may yield noisy or invalid results
due to depth discontinuities, holes, and outliers in the estimated scene.
In DepthLab, we provide two real-time APIs to compute a more stable normal map
in real time on both CPU and GPU. The basic idea is just to sample
a two-ring or even four-ring neighborhood to get a smooth normal map.
Here is the comparison before and after we do the averaging on the normal map. Our algorithm
can effectively smooth the normal map while keeping a reasonable accuracy for developers,
and in DepthLab, we provide a library to cast and reflect virtual laser rays.
With localized depth, we can automatically plan a 3D path for the avatar that avoids a collision
with the statue by making the avatar hover over the obstacle statue.
Particle effects can also be achieved through depth map. For example, each
ring drop tests for hits to hit with the physical environment and render the ripple upon occlusion.
In our work, we forego surface reconstruction and directly represent
environment depth measurements as meshes. This technique relies on a densely tessellated
quad in which each vertex is displaced based on the re-projected depth value.
After initializing the mesh, no additional data transfer between
CPU and GPU is required during the render time, making this method very efficient.
With Surface Depth, we implemented physical collider.
Note that the creation of a mesh collider (mesh cooking) happens
at a lower resolution on the CPU in real time. However, we only perform
it when the user throws a new dynamic object into the AR scene instead of at every frame.
This operation is computationally expensive and not continuously needed
as the physical environment is mostly static. An advanced example is color balloons thrown
onto physical surfaces with texture decals. The balloons explode and wrap around
surfaces upon contact with any physical object, such as the corner of a table.
3D photo effect is also included in DepthLab with texture projection mapping techniques.
When the user click the 3D photo capture button, we save the current camera image,
depth image, and camera parameters. Given the screen center,
we first resolve the central vertex in the real world, based on the current depth map.
Next, we generate a frozen depth mesh. Based on the camera's world position,
we can compute the directional vector from the camera to the central point of the screen.
Next, with a cross product, we compute the plane perpendicular to the directional vector.
Finally, we rotate the camera and recompute the projection matrix
based on the camera's position and project the cached texture to the depth mesh directly.
And you can see the re-projected texture looks like a 3D stereo image effect.
As for dense depth, we introduce a built-in depth-guided antialiasing algorithm to reduce
artifacts due to the low-resolution depth map. The motivation is that closer depth pixels
are typically larger, so we sampling neighboring pixels, we use a larger kernel
size for pixels with smaller depth values. Due to the compute constraints on mobile AR,
we recommend interactions with dense
depth to be implemented on the GPU with compute or fragment shaders.
As for real-time relighting, methods based on BRDFs (Bidirectional Reflectance
Distribution Functions), such as Phong or Lambertian models require a normal map,
which can contain artifacts around object boundaries in low-texture regions (in our case).
In our approach, we compute the photon intensity at the points
when rays intersect with physical surfaces. Please refer to the paper for more detail.
Imagine a ray casting from the light source to the target pixel. The intensity of the photon decays
while traveling along the ray. The intensity is greatly reduced when it hits an obstacle.
And the pixel should look much darker with more obstacles along the ray.
With a little scattering, we can achieve more interesting effects such as
relighting with sun beams.
DepthLab also enables wide-aperture effect, which can be focused on a world-anchored point.
Unlike traditional photography software, which only anchors the
focal plane to a screen point, DepthLab allows users to anchor the focal point
to a physical object and keep the object in focus from even when the viewpoint changes.
For example, just one tap on flower,
it will automatically keep in focus no matter the user approaches close or approaches far.
Of course, dense depth components also included occlusion-aware rendering and fog effects.
Next, we build a minimum viable application to profile a typical usage of DepthLab.
To evaluate the performance of key DepthLab components, we made the application with a point
depth example which is oriented reticle, a surface depth example, which is the depth mesh generation,
and a dense depth example, which is visualization of depth map.
We run our experiments in five locations, each running five minutes and our simple application
runs at over 100 frames per second while surface depth is the most time-consuming part.
In the second test, we evaluated the performance of the real-time relighting,
one of our most computational expensive technique. According to the results, we recommend 4-8 samples
per ray to deploy our relighting module on Pixel 3 or comparable mobile devices.
In the third test, we evaluated the performance of the wide-aperture
effect and recommend a kernel size of 11-21 for real-time performance.
We shared DepthLab with both internal and external partners.
For example, DepthLab components are used in SnapChat and Lines of Play game,
in the Measure App, TeamViewer, SceneViewer, and more. As for limitations, DepthLab is
designed to enable geometry-aware AR experiences of phones with and without time-of-flight sensors.
However, we have not yet explored more in the design space of dynamic depth.
In future, we envision live steps to be available on many IOT devices with camera or depth sensors.
In the future, each pixel in a depth map could be associated with a semantic label and help
computers better understand the world around us and make the world more accessible for us.
Finally, we open sourced the DepthLab library on Github to facilitate
future research and development in depth-aware mobile AR experiences.
We believe that this library will allow researchers, developers, and practitioners
to leverage the base interactions to build novel, realistic AR experiences on regular smartphones.
Finally, I would like to thank all my collaborators including Eric, Max, Luca, Ivo,
Jason, Joao, Jose, Josh, Nuno, Shahram, Adarsh, Kon, and David for their hard work behind the
scenes! And thank you everyone today for listening to my talk! Any questions are welcome! Thank you!