In this video, I’ll introduce AirConstellations, which was done as a collaborative project at
Microsoft Research. AirConstellations is a new technical approach, platform,
and techniques we built to enable flexible cross-device interactions.
It supports a unique semi-fixed style of multi-device interactions, and it does that
by using multiple self-spatially-aware armatures to which users can then easily attach tablets and
other devices. They can then move, re-orient, and pose these devices forming dynamic in-air
formations that suit their current task, social situation, and ergonomic needs.
And this affords an interaction metaphor where relative orientation, proximity, and continuous
movement into and out of these formations can drive context-sensitive interactions.
Now, in the next few minutes, I will give you an overview of the project, introduce our technology,
guide you through a number of application scenarios, and end with a brief discussion.
Our research is motivated by the need for more flexible personal workspaces – at home
or in the office – where people can use their devices working in concert.
The general motivation of cross-device computing is to leverage the complementary roles afforded
by individual devices to support a wide range of application scenarios.
And while many cross-device setups are either fixed or mobile,
we are in particular interested in the third emerging category of semi-fixed configurations.
The goal here is to preserve the modularity and rapid reconfigurability of mobile setups, while
at the same time allowing physical rearrangement of diverse configurations as the task and
situation demand, facilitating people’s rich practices of spatially organizing information.
We designed AirConstellations as a platform to support this semi-fixed style of cross-device
computing. AirConstellations workspaces are composed of 2-5 flexible armatures
with attached devices – such as tablet computers – allowing easy physical movement
and positioning of devices around a workspace. Users can move, re-orient,
and pose devices together to form multi-device assemblages – with devices not only flat on a
table but floating in front, at varying angles, arranged into new workspaces.
We retrofit multiple off-the-shelf device mounts with low-cost sensors
to determine the spatial location and orientation of the armature.
Here we use four IMU and two rotation sensors for each arm, and then combine the tracking
information of each arm into a unified model of device assemblages around the workspace.
To inform the design of this new category of mid-air, spatially-aware cross-device setups,
we synthesized a design space along 16 key dimensions.
These dimensions include characteristics of the spatial configuration – such
as physical arrangement, orientation, structure, or layering – and interactions
behaviors – with dimensions such as motion, direction, or spatial sensing.
To illustrate techniques all across this design space, we implemented
application mockups with a prototyping tool that uses the sensed information of
3D position and orientation of formations to drive implementations of applications
behaviors, responding to any changes of orientation or proximity between devices.
The goal of this prototyping tool was to allow
quick and light-weight experimentation with dynamic application behaviors.
And we were in particular interested in exploring how AirConstellations applications
could dynamically respond to users’ interactions with the formations,
when they move devices in and out of formations, change the orientation of devices, and so on.
We’ll now go through a number of use case applications. And these examples
will illustrate how we use flexible physical arrangement, feedforward of transition options,
and layering of devices in-air across a variety of multi-device app scenarios.
First, our multi-device video conference application allows dynamic reconfiguration
of the task and person space. Views of remote participants can be moved to separate devices,
as physical proxies, resulting in a meeting experience with
spatial gaze awareness and easy access to the shared whiteboard.
The application also facilitates light-weight transitions from group chats to face-to-face
side conversations when breaking one of the remote proxies from the formation.
Participants can use spatial-aware gestures for sharing documents with individuals,
or instead, tilt the device for sharing with the group.
We added feedforward techniques to many of our application scenarios,
for revealing interaction possibilities when moving devices closer together.
In productivity applications, users can distribute tool palettes between devices for easier access,
and application windows can move between devices – either as automatic default behavior in response
to proximity or after manually confirming any of the options revealed through feedforward.
Using physical keyboards attached to AirConstellations allows
proximity-dependent input redirection and moving a tablet next to the keyboard can
trigger context-dependent soft-keyboard extensions.
In another example, for data analysis, users can quickly assemble multi-display
arrangements, without the need to manually change system settings,
and they can change the configurations when needed.
They can also form larger interactive horizontal surfaces by moving devices together. And here,
feedforward animations indicate possible formations of devices,
and once joint all devices can be used like a single continuous surface.
Breaking up device formations by moving a single device allows the direct transition
from shared to personal interaction, for example for overview+detail techniques.
Many other applications are possible. For example, AirConstellations can
provide orientation-aware windows and semantic zoom of controls in a flight simulator game.
And in a companion setup together with a digital whiteboard,
we can use device proximity to open ad-hoc portals for direct transfer of sketches.
We also explored system-wide cross-device behaviors.
For example, when devices physically overlap, see-through portals across
desktops can facilitate joint access and transfer of files or applications.
And layering of devices can facilitate cross-device window management
and light-weight application migration through selecting from a semi-transparent preview.
Let me close with a brief reflection on the potential of dynamic in-air formations,
as well as limitations and possible next steps.
allows using proximity and location as an implicit parameter for fluid interaction techniques. We
leveraged this in several of our techniques, such as feedforward. At the same time, such automated
tracking – and the technical overhead it brings with it – might not always be needed or desired.
Second, it is also important that we design for easier transitions between fixed, semi-fixed,
and mobile configurations. With ever-changing workspaces, home-office locations, and dynamically
evolving device ecologies, it becomes critical to best support such ad-hoc working patterns.
Third, we have to consider the tradeoffs of proactive application behaviors.
As we showed, sensing the proximity, relative orientation, or adjacency of
displays affords certain automatically sensed transitions in application states or behaviors.
However, people’s natural socio-spatial literacy implies that such device motions
and configurations are also already part of the vocabulary of human-human
interaction. So we have to find techniques addressing this potential design tension.
Fourth, we can explore other use cases and form factors of devices and armatures.
For example, dynamic in-air device formations could be applied in AR
or VR applications, in education, or in specialized domain expert workspaces.
In summary, we introduced AirConstellations which afford
highly flexible and dynamic device formations Users can bring multiple devices together
in-air — with 2–5 armatures poseable within a workspace — to suit the demands of their
current task, social situation, app scenario, or mobility needs.
AirConstellation devices, in combination with appropriate applications, feedback,
and interaction techniques, open up an intriguing new design space.
The scenarios we demonstrated represent only a subset of this design space
of highly dynamic and easily reconfigurable personal workspaces with in-air device formations,
and we hope our work will inspire many promising research avenues to explore.