Greetings, travelers!
I am Ruofei Du from Google.
Today, I am going to present our open sourced magic: DepthLab: Real-time 3D Interaction
with Depth Maps for Mobile Augmented Reality.
Let's enjoy a short video first.
DepthLab encapsulates a variety of depth-based UI or UX paradigms including geometry-aware
rendering, real-time visual effects such as rain, snow, flooding, and re-lighting, and
surface interaction behaviors such as surface splatting, real-time mesh generation, and
physics simulation.
DepthLab is available on a large number of Android devices with a single RGB camera.
We compute live depth map by the depth from motion algorithm introduced in SIGGRAPH Asia
2018.
However, there is a large gap between this raw depth data and the typical expertise of
mobile application developers who are not experienced in handling depth data.
In DepthLab, we process the raw depth map from ARCore Depth API and provide customizable
and self-contained components for mobile AR developers.
To explore the design space, we conducted three brainstorming sessions with a total
of 18 participants and proposed a total of 39 aggregated ideas.
We architect and implement a set of data structures and real-time algorithms for mobile AR developers.
There are three kinds of data structures:
First, depth array, which stores depth in a 2D array of 16-bit integers on the CPU.
Second, depth mesh, a real-time triangulated mesh generated from depth map.
Third, depth texture, a GPU texture decoded from the depth array.
Based on the data structures, we classify DepthLab components into three categories:
localized depth, surface depth, and dense depth.
Localized depth uses the depth array to operate on a small number of points directly on the
CPU.
For example, we provide a 3D oriented cursor to convert any screen point to a world point
and orients according to surface normal vectors.
With a smoothed normal vector estimation algorithm, DepthLab provides a sample SDK to cast and
reflect virtual laser rays.
DepthLab can also automatically plan a 3D path for the avatar that avoids a collision,
by making the avatar hover over the obstacles.
As for particle effects, each raindrop tests for a physical collision and renders a ripple
when reaching any surfaces.
In surface depth, we forego surface reconstruction and directly represent environment depth measurements
as screen-space meshes.
This technique relies on a densely tessellated quad in which each vertex is displaced based
on the re-projected depth value.
With surface depth, we implemented a physical collider.
An advanced example is color balloons thrown onto physical surfaces with texture decals.
3D photo effect is also included in DepthLab with texture projection mapping techniques.
As for dense depth, we introduce a depth-guided antialiasing algorithm to reduce artifacts.
Relighting is an important use case of dense depth.
We cast rays from each light source to each pixel and estimate the photon intensity arrived
at the pixel in real time.
DepthLab also enables wide-aperture effect, which can be focused on a world-anchored point.
Unlike traditional photography software, which only anchors the focal plane to a screen point,
we allow users to anchor the focal point to a physical object.
For example, just one tap on the flower, it will automatically keep in focus no matter
which viewpoint the camera observes the flower.
Of course, dense depth components also included occlusion-aware rendering and fog effects.
DepthLab components are used in SnapChat, Lines of Play,
the Measure App, TeamViewer,
and more.
Although DepthLab is designed to enable geometry-aware AR experiences on phones with and without
time-of-flight sensors. we have not yet explored more in the design space of dynamic depth.
Finally, we open sourced DepthLab on Github.
to allow researchers, developers, and practitioners to leverage the base interactions to build
novel, realistic AR experiences on regular smartphones.
Finally, I would like to thank all my collaborators including Eric, Max, Luca, Ivo, Jason, Joao,
Jose, Josh, Nuno, Shahram, Adarsh, Kon, and David for their hard work behind the scenes!
And thank you everyone today for listening to my talk!
Any questions are welcome!
Thank you!