Hi. Our names are Frederik Brudy and David Ledo. And we are very excited to be presenting
our work on SurfaceFleet here. We did this work during an internship at Microsoft Research,
together with our co-authors. SurfaceFleet is a system and toolkit that explores an interaction
paradigm for migrating content and tools across devices, applications, users, and time. Given
technological advances, knowledge work has become dynamic, spanning a variety of transitions
from one “place” to another. These transitions can be in the form of a physical location,
migration of work across devices, the presence or absence of a collaborator, or the availability
of a particular piece of information at a given time. In contrast, current devices consider
place in a more general manner, where a place is a physical location and most often tied
to a single user, ultimately confining interactions to a single user within a single application.
Therefore, transitioning digital content beyond these confines comes at a high cost of both,
application development and user experience. To enable mobility of work beyond a single
place, we created SurfaceFleet. SurfaceFleet is a system that addresses these challenges
by de-coupling user interface elements and operations from a particular device, across
applications, for single and multiuser collaboration, as well as at the time that suits best. Our
contribution is part design probe, part reference implementation, and part systems proof-of-concept.
On the UI side, SurfaceFleet utilises Applets, which are semi-transparent UI elements that
float on-top of the existing desktop environment and software. Because the Applets are always
readily available, it becomes possible to work across applications, synchronously or
asynchronously. As a result, SurfaceFleet proposes an alternative way to think about,
design, and implement distributed interactive systems. SurfaceFleet Applets unbind interactions
across four dimensions. First, Device. Applets run on individual devices, but their underlying
state is preserved through our framework via the Azure cloud. As a result, applets, as
well as their states and properties can be synchronized across multiple devices. Second,
it unbinds interaction from Applications. We enable tool and content reuse independent
of the application. Applets float above the application windows, and conceptually above
the device itself. Therefore, people can operate Applets to interact with the underlying applications
and operating system, regardless of app manufacturer. Third, User. Since Applets float above the
device—and their content can be synchronized—it is possible to share Applets to further empower
individual, as well as collaborative work. And lastly, with SurfaceFleet we unbind interactions
from Time. By having a robust state preservation, as well as sharing of content and tools, we
consequently support synchronous and asynchronous actions. And these four dimensions work in
tandem, allowing users to selectively share content and tools, or defer actions until
the desired resources are found. It becomes possible for people to interact with digital
content both individual and collaboratively at the right place and the right time. In
terms of previous systems, SurfaceFleet builds on large range of systems and techniques for
interactions across devices, applications, and individual and collaborative work—or
for deferring actions in time. Our work falls into the area of cross-device interaction,
where SurfaceFleet supports one or many people and devices, synchronous and asynchronous
interactions, fixed and ad-hoc dynamics, personal and social scales in a remote or co-located
manner. Previous systems have explored interactions that span across multiple devices in a single
user context. Or they leveraged external sensing to operationalise f-formations and micro-mobility
in a collaborative cross-device interaction. Such systems often utilise simple network
protocols via UDP or TCP to communicate between devices allowing prototypical explorations
of interaction techniques, or by using a shared dictionary. But also shared Document Object
Model is being used for such systems, such as with. However, this limits the interaction
to the walled garden of the web and does not allow for cross-application interactions.
With SurfaceFleet we are building on a robust system foundation, that offers resilient logging
and recovery of data. As a result, this allows to transition fluently between devices, but
also unbinds interaction from application, user, and time. And while these 4 dimensions
individually have been explored in prior CSCW, Ubicomp, and cross device research with SurfaceFleet
we provide a unified interface around a number of concepts to bridge devices, applications,
users, and time. And we are addressing a variety of cross-device usage scenarios that entail
crossing one or more of these four bridges. So let’s look at some of the techniques
supported by SurfaceFleet. SurfaceFleet lives in the taskbar, where it is possible to create
and access all available applets. As said, applets are visual representations of the
content ant tools, that float on top of the users’ Desktop. An example of such an applet
is a media primitives. Media primitives are pieces of content, such as images, which can
be selected for example from an existing file or from a camera. Media Primitives can also
act as placeholder, to be fulfilled at a later time. Media Primitives can be dragged directly
into individual applications. Containers are a collection of Media Primitives and Tools.
They are like floating stacks of paper, and the contents are displayed as a sheaf on the
user’s screen. Containers can be shared across a user’s devices and between different
users. Containers then synchronise the content. Users can operate tools across applications
to create a seamless workflow. For example a color picker that goes from one device to
another to grab reference material. These tools can be shared through portfolios to
foster collaboration. For example in a group setting participants can share their mouse
tele-pointers and keyboards to operate a common large display. We also have "Portfolios",
which are part artist's folio and part portal. Users can create “Portfolios” in their
own style and it fades into the background when not in use. Portfolios enable sharing
of different media primitives simply through drag & drop. One can also share containers
as well as tools. Portfolios can reside on multiple devices, enabling both individual
transitions, as well as group collaboration. For example you can ask a colleague at a construction
site to share their camera feed through the portfolio. You can watch live, even from multiple
colleagues at the same time, extract images, and use them in your local documents. Interaction
Promises are placeholders for content for fulfillment at a later time. Interaction Promises
offer novel behavior by acting as a reference for future content, such as an on-site photo
that has not yet been taken. Through interaction promises, it is possible to defer actions
or delegate tasks across collaborators. For example, in Word. Just create a placeholder
and pass it along to a colleague. Then, they can provide the missing information—such
as a photo needed from their site visit—whenever they take the picture on-site. Or you can
change your mind later, and swap the picture for another—the Promise links back and is
automatically updated. Similarly, you can insert a Container Promise, collect images
from multiple collaborators, and then select your favourite to use in your document—or
try out different ones until you find the best fit. SurfaceFleet runs on Windows and
is implemented in C# using .NET Framework and Windows Presentation Foundation (WPF).
Major components include a shared model, robust logging of updates, and OS and application-interop
features. In terms of code it can be as easy as tagging a property as "Synchronizable"
to make it shared across different devices. Our toolkit now enables two ways of synchronizing
data: through a state model which preservers the data in a persistent manner and synchronises
it across multiple devices. Second, we have transient data that is not state preserved,
but we use for ephemeral data, such as sensors which are only available and relevant at a
certain time. Each individual client now synchronizes and resynchronizes application state—essentially
turning a transition from one device to another into a cross-device fail-over. As mentioned
before, SurfaceFleet is not siloed within our own ecosystem. The applets communicate
with the underlying Windows system as well as other third-party applications in 4 different
ways: First, through programming APIs we can access operating system functions such as
the clipboard or keyboard events, and through COM APIs we can communicate with other applications.
Second, we use the Windows clipboard to easily exchange data with external applications.
Third, we can also implement custom plugins for even deeper integration into external
applications. Lastly, we leverage the multi-format support of the Windows drag & drop events
to enable exchange of data within our application but also exchange data with external apps.
Let’s spend a bit of time to explain how this works in practice. And we’ll show this
with the example of how we implemented Interaction Promises. When starting a drag & drop operation
with a media primitive, we place information about that primitive in in the drag event’s
data attribute using multiple formats at the same time: a standard image but also data
containing an internal reference. When dropping the primitive onto Word, we then use Win32
APIs to detect which application the media primitive is dropped on. If we recognize the
application, we can then use its public COM APIs. In our example here, we use the Word
APIs to insert a bookmark, which encapsulates the placeholder image as well as a hidden
unique ID. Later, when the Interaction Promise is fulfilled, we can scan the Word file for
that bookmark again and simply replace the image. And I want to emphasize that this all
works without modification of Word or Windows code. In fact, we similarly integrate other
applications, such as Adobe Illustrator and Photoshop. We have a few discussion points
that are worth sharing. SurfaceFleet looks at how knowledge workers transition from one
place to another, where “place” is shaped by our 4 dimensions of unboundedness. The
consequence is that sharing is no longer limited to files and folders, but further expands
to applets and tools. Now as a result we can say that sharing is about pieces of documents,
about actions and interactions, and other intermediate work-objects. And all these objects
have an associated visual representation that can be easily understood more more closely
matches the task that is being done in the real world. Together with Interaction Promises
this makes for a unified interface that can support flexible workflows and different orders
of operation. We also briefly want to talk about scaling of our concepts in Time and
Number. SurfaceFleet is intended for transient work objects in active use. It is not intended
for, nor designed for cold storage of files, or for having a lot of applets on the desktop.
Indeed, having a lot of applets can lead to a “messy desk” which may not necessarily
scale at large. However, we still believe that our four cornerstones of unboundedness
from device, application, user, and time still hold truth. And that they can be used as a
lens to further explore other design choices. Some people might now say that multi-device
file synchronisation has been long solved, for example with Dropbox and OneDrive. And
similarly that remote collaboration has been solved with Teams, Slack, Google Docs, and
Microsoft 365. And I can always open an earlier version of a file that was posted on a Slack
channel, travel back in time on OneDrive, or use versioning in Git. However, we argue
that with SurfaceFleet we address all of these four cornerstones at once. In contrast to
existing file sharing applications, we focus on transient pieces of content or tools in
active use. These pieces of information are always available, and not siloed within an
application’s message thread. We enable single user as well as multi-user interaction
through the same means, by allowing tools as well as media primitives to travel between
device boundaries, no matter who owns those devices. And through Interaction Promises
we enable asynchronous work that is not file centric, that is not bound to any particular
application, and that is deeply integrated into content. Through SurfaceFleet, it becomes
possible to break siloed content out of their applications and go beyond the boundaries
of a single device. The techniques that we presented—in the form of tools, inputs,
content, and behaviors—offer a variety of distributed user operations that enable delegation,
deferred actions, and even unstructured workflows. Moore’s law is at an end, given that computational
power is now longer growing exponentially. However what is growing exponentially is storage
and network bandwidth. Which means, that perhaps the future is more about the emphasize mobility
of data, rather than any particular CPU. At the same time there are more and more connected
devices with and around us, including tablets, laptops, mobile, large displays, and so on.
Which means, that the future is more about the mobility of user activity, rather than
any particular device. And we believe, that all this hints at a Society of Technologies
that establishes meaningful relationships amongst the members of this society, where
interactions are unbound from device, application, user, and time. Thank you so much for attending
our talk.