(electronic music)
- Hi, everyone, my name is Dan Bohus.
I'm a researcher in the Adaptive Systems
and Interaction Group at Microsoft Research in Redmond.
I joined Microsoft Research in 2007,
and pretty much ever since my research has focused
on issues of physically situated interaction,
basically how can we get computers to reason more deeply
about their surroundings, understand how people behave
and interact with each other in physical space,
and participate in these interactions in a fluid manner.
Now, over the years, we've studied a number of problems
in this space, and develop a variety
of prototype interactive systems,
from virtual assistants that live outside your office door,
and interact with people that come by
when you're not present to perhaps negotiate
some calendaring tasks on your behalf,
to robots that give directions in the building
with coordinated pointing, and gaze, and gesture,
and speech, all the way to elevators that
recognize intentions to a board, or walk by.
These type of systems that a reason about
and act in the physical world are generally multimodal
and integrative in nature.
That means they leverage many different kinds
of streaming data and bring together different kinds
of AI technologies to solve an end-to-end problem.
For instance, in my own work, I've been very interested
in how physically situated language interaction works.
How can we get computers to understand
the finely coordinated dance that we do with each other
when we interact with spoken language
in face-to-face settings, with all the coordination
that happens in our verbal and non-verbal behaviors?
Clearly, to construct interactive systems like this,
we need to leverage data from both the audio
and visual domains, and we need to bring to bear
a lot of different AI technologies.
Now, over the last decade,
we've seen a lot of significant progress
with machine learning techniques and large data sets
on a number of core important AI problems
in computer vision, speech recognition,
a variety of natural language processing tasks.
And this is very exciting,
because these perceptual technologies are basically
the core building blocks required
to get computers to start to understand the physical world.
Now, despite the progress that we've made though,
putting these technologies together,
and constructing end-to-end systems that
operate seamlessly in the real world
remains a very challenging task
from an engineering standpoint,
even if we leave aside all the scientific challenges
that might arise in any specific application.
And I would argue that a lot of these engineering challenges
stem on one hand from the sheer complexity of the systems,
as we said, they use many different kinds of components,
and on the other hand from the fact that
the infrastructures and programming fabrics
and the tools we use to develop the systems today
are not very well attuned to the needs
of these applications.
As I mentioned before, the systems that reason about
the world generally process multimodal data streams
in real time, so images, audio,
depth maps, LIDAR information.
This data comes streaming with different bandwidth
and different latency requirements.
The processing often involves a heterogeneous set
of components from sensor-specific,
as the case in drivers, to machine learning models
that trigger large GPU loads to cloud services,
to integration with application-specific backend logic.
Now, for these kind of systems that operate in the world,
time is also very important.
And so, the execution of these components needs to be
very finely coordinated and done in a very efficient manner
because the systems often operate
under important latency constraints.
And oftentimes, when you build the systems,
you find yourself developing
low level, you know, synchronization,
or fusion problems, or optimizing something,
or building logging infrastructure,
instead of really focusing on the task at hand.
So over the years, we've encountered
many of these challenges firsthand,
and I believe that anyone that has tried
to build one of these multimodal, real-time,
integrative AI applications has probably seen them as well.
And so, based on all these lessons learned,
as well as experiences people in our team have had
in robotics domains, we have developed
at Microsoft Research a framework that aims
to alleviate these engineering challenges,
and accelerate research and development
for these multimodal integrative AI applications.
The framework is called Platform for Situated Intelligence,
and it is open source and available online
at github.com/microsoft/psi.
So Platform for Situated Intelligence is built
on top of .NET.
So it runs cross-platform on both Windows and Linux,
and it retains the modern affordances
of programming languages like C# and F#,
so things like type safety and memory management,
while addressing the very specific needs
that multimodal integrative AI applications have.
Concretely, the framework provides affordances
in three different areas,
infrastructure, components, and tools.
With respect to infrastructure,
the Platform for Situated Intelligence runtime
provides programming model for working
with multimodal, temporally streaming data,
in which applications are composed
by connecting components together with streams of data.
Time is a first-order object in this framework.
Messages that flow through the processing pipeline
are timestamped at the source
when they are acquired from the world
with something we refer to as an originating time.
And this originating time automatically carries
through downstream in the entire computation pipeline,
enable latency awareness,
as well as correct and reproducible synchronization
and data fusion anywhere downstream.
This is a very important property in the systems.
And apart from this, the infrastructure provides
a number of other benefits,
including data logging and data replay,
component isolation, remoting, and so on.
For instance, data streams can be easily logged to disk,
and then resurfaced from disk later,
simulating a live execution,
which enables a data-driven development cycle,
where you repeatedly experiment over the same data,
and tune, and improve your components over time.
So apart from the infrastructure,
the framework comes also with an open growing ecosystem
of components that wrap various sensor technologies,
like microphones, cameras, Azure Connect device,
and various processing technologies for processing images,
and audio, speech language,
running pre-existing machine learning,
ONNX models, or connecting to various cloud services.
The basic idea is that to write an application
you would hopefully find a lot
of available off-the-shelf components,
and you connect them together,
and perhaps you write a couple of components
that are specific to your particular application.
So for example, in one of our introductory demo samples
we have in the repo, we instantiate an Azure Connect camera.
We perform body tracking based on the Azure Connect.
We detect when the person raises their arm and points,
and compute the direction in which they're pointing.
We intersect that direction with a depth mesh,
and take that orange intersection point
and project it back into the RGB camera
of the Azure Connect sensor.
We crop an image patch around that projection point,
and send that up to Azure Computer Vision service
to do object detection.
In parallel with all of this,
we capture data, audio from a microphone,
and segment it using voice activity detector,
and perform speech recognition,
again, using cloud services,
the idea being that you can walk into a room,
point to an object, say, "What is that,"
and the computer understands what you're pointing at.
Now, this is just a small sample application.
It's a couple of hundred lines of code.
It's available in the repository.
And in reality, applications developed with this framework
can grow significantly more sophisticated than this,
but this is just meant to illustrate the ease
with which you can connect a variety
of different components together,
from streams coming from Azure Connect sensors,
to cloud services for computer vision
and speech recognition, to locally performed computations
to detect when someone is pointing,
and the direction they're pointing into, and so on.
Now, when you write these type of applications,
another big part of development has to do
with debugging, maintenance, and iterative improvement.
The framework provides a companion tool
called Platform for Situated Intelligence Studio,
which allows developers to visualize streams of data
persisted by applications written using this framework.
It's very easy to construct complex visualization layouts
that involve timeline visualizers for numerical data,
and audio, and so on, 2D visualizers,
things like images, object detection results,
3D visualizers, point clouds, track bodies, camera views.
And it's very easy to inspect these data.
You can do so over time.
You can navigate easily over time,
and inspect what happened in the application
at different time points,
and this can significantly accelerate the debugging
and maintenance process.
The visualizations can be done not only offline
based on persisted data,
but also live, while the application is running.
Also, apart from visualization,
the tool supports data annotation and batch processing,
which are, again, constructs that are very important
when you're dealing
with these data-driven development loops.
Altogether, we believe this set of tools,
together with the infrastructure provided by the runtime
and the set of components available,
can significantly accelerate work
for multimodal integrative AI applications.
So in case you think this might be interesting
for your own work, please check it out.
As I mentioned, the framework is open source,
available at github.com/microsoft/psi.
We have a lot of documentation,
which is hosted in the wiki pages on the GitHub site.
And on the right hand side,
there's a sidebar that essentially provides an index
into the various documentation topics.
The best place probably to get started
learning about the framework
and coding with Platform for Situated Intelligence
is the brief introduction tutorial
that introduces some of the main concepts step-by-step.
I would also recommend probably perusing
some of the samples that we have in the repo.
A lot of these samples have detailed walk-throughs
that walk you step-by-step through how the sample was built,
and point to more in depth documentation in relevant places.
The samples vary in complexity.
They start with simple ones that just illustrate
how do I construct the first pipeline,
and create the first stream,
to how do I capture, you know,
audio and video data from a webcam and a microphone,
all the way to the more sophisticated example
I just described, where you're able to point to objects
in a room, and the machine understands
what exactly you are pointing to.
Apart from the GitHub repo, a recent blog post
that provides a high level overview of the framework,
and also a webinar that's available online,
which apart from the overview of the framework
also contains a 30-minute hands-on coding tutorial,
how to get started with Platform for Situated Intelligence.
Platform for Situated Intelligence is the result
of the work of a team of talented engineers
and researchers at Microsoft Research,
and I wanna thank and acknowledge them here.
So we're looking forward to engaging
with a broader community,
and growing this open source effort.
We encourage you to use the framework, and contribute.
No contribution is too small,
from simply filing issues when you find bugs,
and giving us feedback,
to developing your own components,
and contributing them, or releasing them yourself,
to developing entirely new features.
We look forward to hearing from you,
and we hope Platform for Situated Intelligence
will accelerate your work.
(electronic music)