Phonetroller is a low-cost system to use mobile phones as touch-enabled VR controllers
with visual feedback of the operating hand to facilitate precise touch input.
Hand capturing is achieved without external cameras or advanced phone sensors.
The system consists of a mount attached on the phone
with a mirror above the screen that reflects the front-facing camera.
A VR tracker is added on top of the mirror to track the device in 3D space.
Capturing is performed by the front camera through reflection,
with a green screen and 2D markers allowing speedy
and robust segmentation of the user's thumb moving above the screen.
This thumb mask or shadow can then be overlaid
as a semi-transparent texture on a virtual touch interface,
such as the screen of a virtual phone,
to help precise aiming of touch targets.
Additionally, a deep neural network inspired by a U-Net
can be used to infer the position of fingertips
above the screen from the mask image.
This allows input and feedback based on hover points
without relying on other sensors.
Visual pre-touch feedback can be used to operate mobile phones
in VR applications similar to remote controls.
In a block-modelling application,
the phone is used as a pointing device to select blocks
and a touch-based toolbox to activate tools.
The compact toolbox interface is shown as a heads-up display
in a corner of the viewport
from which tools can be selected
by tapping buttons with help from the thumb overlay.
The toolbox can become a numeric keypad
to enter specific numeric values for scale,
position or other parameters.
The presence and absence of the thumb in the camera's view
can be used as condition to show and hide the HUD.
This allows the user to summon the tools only when desired
and otherwise keep a full open view of the VR space.
The block-modelling tools can be used for another purpose:
Annotation of point clouds.
In this version,
transparent 3D blocks are positioned and reshaped
to enclose points belonging to a specific object to annotate,
for example a computer.
An annotation can then be chosen from a list of existing labels,
or, a custom label can be entered manually using a touch keyboard.
The user can change the location of the toolbox UI
by grabbing it with the phone in an upright position,
moving it to another location
and tilting the phone back to disengage.
Bringing the phone close to the headset
switches to a centre-screen HUD
with only icon outlines.
This HUD style helps to simultaneously focus on content and tools.
The application also supports the use of a second controller.
In this bimanual configuration,
the user points with the controller
and selects tools with touch on the phone.
This clearly separates the roles of pointing and touching
but also requires hand coordination.
Hand shadows can also facilitate sketching with a pen.
In this application,
users doodle with a stylus on the phone
and then use it as a rubber stamp
to paste their sketch on a virtual wall.
The overlay mask can also be used to identify pen from finger touches,
for instance to switch to an eraser.
Note again that no external marker or tracker is required to track the pen
so that any digital stylus can be used in this setting.
Finally,
Phonetroller can be used purely as a touch device
to enable mobile gaming augmented with VR.
In this setting,
the phone is held with both hands like a handheld console
and the touch overlay spans the entire viewport.
In this game example,
one hand or thumb is used to invoke a menu
and perform actions on content located under the gaze crosshair.
The other thumb controls targeted actions
like wielding a sword in specific directions.
The second thumb can also be used as an alternative pointing method:
Using the hover point inferred by deep learning,
objects can be selected and highlighted by hovering over them,
similar to a mouse cursor.
In a user evaluation of targeting speed and precision with two target sizes,
thumb shadows on a virtual phone and a fixed HUD
were compared to three baseline conditions:
hover points obtained from the capacitive sensor of the phone,
a Samsung Galaxy S5,
an indirect confirm-on-release input method with no hover feedback,
and normal phone use without VR.
The main finding is that targeting with the thumb shadows
is significantly faster than with hover points sensed by self-capacitance
and beats indirect input for fingernail-sized targets.
In summary,
Phonetroller is a low-cost, low-engineering solution
to support precise touch input on mobile phones in virtual reality.
For further details please refer to the paper.