Did you ever wonder what it would take for mobile devices
to use their camera to understand their surroundings,
like we do?
Project Tango is a computer vision platform
that can do that by tracking motion, recognizing
spaces, and scanning the surroundings in 3D.
My name is Nicolai and in this episode,
we will talk about how motion tracking gives mobile devices
the power to track fine grain movement and move with you.
Motion tracking can follow your motions smoothly,
whether you're standing, walking or biking.
We found that it works well on earth,
but we took the testing one step further.
Project Tango was launched to the International Space Station
as part of the SPHERES robots program.
These were self-propelled robots that floated around the space
station, and Project Tango was integrated
to give live motion tracking feedback.
In this episode, we are going to share with you
how motion tracking works and how
you can get started with it.
Imagine you had a virtual solar system in your living room.
With motion tracking, you could simply
walk from planet to planet to have
a closer look at each of them.
Or you can use the device to control a spaceship as it
descends and flies around on the surface of one of the planets.
Whether you're building a space game or something else,
you can use motion tracking to take the user on a journey
through a virtual world.
So how does motion tracking work?
Project Tango collects data from a fisheye camera
and an inertial measurement unit, or IMU.
The image from the camera is used
to identify visual features, such as edges and corners,
and track how much they move between frames to determine
distance traveled.
This is called feature tracking.
The IMU uses an accelerometer and a gyroscope
to track how fast a device accelerates
and in which direction it's turning.
Lastly, the images are fused with the [INAUDIBLE] IMU
sensor data to calculate how much the device has moved.
Did I mention that this all happens
on the device, in real time, and up to 100 times per second?
And since 60 images per second is why motion tracking looks
smooth to the human eye, users get a seamless sense of motion.
Now, one challenge is that the human eye has a very wide field
of view, almost 180 degrees, which
greatly improves our ability to see and run away from tigers.
So in order to increase motion tracking accuracy
in mobile devices, Project Tango uses wide angle fisheye lenses
at up to 160 degrees to capture details of the environment.
Think about the corner of a doorway.
It has high contrast and can be easily identified over time.
For motion tracking to work well,
we need to identify many of these features
in the environment.
And the wider the camera angle, the better
our chances to find more such features,
which in turn increases the reliability and accuracy
of motion tracking.
In order to use motion tracking, you
need to understand Project Tango's coordinate systems
and the concept of frame of reference.
Project Tango tracks the relative position
of the device in three dimensions
from a starting point.
This point is called the origin, and it's always zero-zero-zero.
The orientation of the device is also
tracked around the same three axes.
Since we can move and rotate on each axis,
we end up with six degrees of freedom in total.
Together, we refer to the combination
of precision and orientation as the device post.
The other important concept is frame of reference.
In motion tracking, everything happens
relative to where you start.
This is the origin.
The origin always coincides with where you connect to and start
your Project Tango service.
We refer to the frame of reference
as the start of service frame.
The start of service frame in turn
relates to the frame of the device.
When you combine the two frames, you
get the position and orientation of the device in space.
Project Tango supports many more frames of reference,
but for motion tracking, this is all you need.
Let's have a look at Project Tango Explorer
to see how motion tracking works in practice.
We are going to use Explorer in its default mode, which shows
the raw motion tracking data.
The pyramid, also known as the device frustum,
displays the orientation of the device,
and the app shows the position as a blue trace
when you move around.
If we walk in a circle, the device
will show a circle on the screen.
If we draw a circle with the device in front of us,
you will see it works in all three dimensions.
It is fundamental for us to be able to move,
yet currently available devices only
track orientation and position at a coarse level.
Motion tracking in Project Tango is
all about giving mobile devices an understanding of movement
that fully tracks the way we move.
Altogether, it lets the device become a window
into a virtual space, enables augmented reality,
and makes it possible to use the device as a motion controller.
While we continue to improve the robustness of the system,
APIs are available so you can take
advantage of motion tracking and build truly mobile experiences.
Visit our Google+ community and join us on our journey.
We are excited to see what you will build with Project Tango.