AirConstellations is a new technical approach, platform, and techniques that enable flexible
cross-device interactions where users can move, re-orient, and pose devices together
to form dynamic in-air device formations.
Physical changes are sensed by multiple spatially-aware, self-stable armatures that operate within
the same working volume.
Relative orientation, proximity, and continuous movement into and out of ad-hoc device formations
can afford natural transitions of content and apps across devices.
Our research is motived by the need for more flexible personal workspaces – at home or
in the office – where people can use the devices they own and use in concert.
Cross-device computing can leverage the complementary roles afforded by individual devices to support
a wide range of application scenarios.
AirConstellations workspaces are composed of 2-5 flexible armatures with attached devices
– for example tablet computers – allowing easy physical movement and positioning of
devices around a workspace.
We retrofit multiple off-the-shelf device mounts with low-cost sensors to determine
the spatial location and orientation of the armature.
We use four IMU and two rotation sensors for each arm, and then combine the tracking information
of each arm into a unified model of device assemblages around the workspace.
We synthesized a design space for mid-air, spatially-aware cross-device setups, along
14 dimensions of hardware configuration – such as physical arrangement and orientation, structure,
or layering – and interaction – with dimensions such as physical reach, approach of devices,
sensor mapping, or behavior.
We will go through a number of applications and interaction behaviors, illustrating techniques
all across the design space.
All applications are implemented with a prototyping software that uses the sensed information
of 3D position and orientation of devices and formations to drive mock-up implementations
of applications behaviors, responding to any changes of orientation or proximity between
devices.
First, our multi-device video conference application allows dynamic reconfiguration of the task
and person space.
Views of remote participants can be moved to separate devices, as physical proxies,
resulting in a meeting experience with spatial gaze awareness and easy access to the shared
whiteboard.
The application also facilitates light-weight transitions from group chats to face-to-face
side conversations when breaking one of the remote proxies from the formation.
Participants can use spatial-aware gestures for sharing documents with individuals, or
instead tilt the device for sharing with the group.
We added feedforward techniques to many of our application scenarios, for revealing interaction
possibilities when moving devices closer together.
In productivity applications, users can distribute tool palettes between devices for easier access
and application windows can move between devices – either as automatic default behavior in
response to proximity, or after manual confirming any of the options revealed through feedforward.
Using physical keyboards attached to AirConstellations allows proximity-dependent input redirection
and moving a tablet next to the keyboard can trigger context dependent soft-keyboard extensions.
For data analysis, users can quickly assemble multi-display arrangements, without the need
to manually change system settings, and change them when needed.
Larger interactive horizontal surfaces can be formed by moving devices together.
Feedforward animations indicate possible formations of devices, and once joint all devices can
be used like a single continuous surface.
Breaking up device formations by moving a single device allows the direct transition
from shared to personal interaction, for example for overview+detail interaction.
Many other applications are possible.
For example, AirConstellations can provide orientation-aware windows and semantic zoom
of controls in a flight simulator game.
And in a companion-setup together with a digital whiteboard, we can use device proximity to
open ad-hoc portals for direct transfer of sketches.
We also explored system-wide cross-device behaviors.
For example, when devices physically overlap, see-through portals across desktops can facilitate
joint access and transfer of files or applications
and layering of devices can facilitate cross-device window management and light-weight application
migration through selection from a semi-transparent preview.
In summary, AirConstellation devices, in combination with appropriate applications, feedback, and
interaction techniques, open up an intriguing new design space of highly dynamic and easily
reconfigurable personal workspaces with dynamic in-air device formations.