(bright upbeat music)
Hi everyone.
My name is Weishung Liu and I'm a Senior Program Manager
at Microsoft Research in Redmond.
My team works on computer vision, low cost AI infrastructure
and distributed systems.
Specifically, we work on project Watch For,
which is a large scale, low cost, highly programmable
video analysis platform.
In the last five years, video has become ubiquitous
and permeates many aspects of our lives,
from entertainment, to education and communication,
we generate years worth of video every day.
But how do you find that critical moment
in the sea of frames?
How many times have you started watching a recording
of a meeting or of a lecture,
and you just can't find that specific meeting topic
or that really important action item
or that key explanation?
Or maybe someone has been live streaming for over 12 hours,
but you only wanna watch
when the content is most interesting.
Here's where the Watch For team came in.
We started from humbled beginnings
as the grand prize winners of the 2017 One Week Hackathon.
At the time Lenin Sivalingam, Peter Bodik
and Matthai Philipose developed a method
to apply computer vision, to live streams,
to automatically identify interesting events in real time.
They took a unique approach to solving this problem
by focusing on three key dimensions.
Programmability, cost, and scale.
Watch For is designed to be highly customizable
and adaptable to any business need.
Unlike existing solutions,
developers can specify custom logic
for processing incoming video frames
in a video AI application
tailored to their specific use cases
without sacrificing cost, efficiency, or scalability.
In fact, applications can mix and match
multiple models concurrently,
which can include a combination of pre optimized models
from the Watch For library
and models trained in Watch For's model training platform.
Watch For uses a combination of different techniques
in different parts of the pipeline
to provide significant savings.
The techniques can be broadly classified into three buckets.
The first is around resource utilization.
We try to make sure that network and CPUs
are effectively utilized.
One way that we do this is,
by dynamically finding the optimal processor
and thread configuration for each workload.
Most systems are set up to have end threads per process
and apply that configuration to everything.
On the other hand, the Watch For system
enables developers to test and visualize
different configurations
so that they can find the optimal utilization
for their unique workloads.
We try it in cache and reuse intermediate results.
We also do machine learning optimizations like batching,
model cascades, and even model optimizations
such as quantizing them to low bit widths.
Finally, we expose efficient programming templates
in as many knobs to the developer as possible.
So that optimizations such as adaptive sampling
and de-duping can be easily implemented.
This has enabled Watch For to be significantly cheaper.
Watch For is designed
and optimized to support video and livestream workloads.
Many machine learning optimizations
are focused on small record data like speech,
text and images.
But, when you start executing models on videos,
resources can take a substantial hit in memory,
network and compute usage.
We designed the platform in a way that makes it easy
for any developer to ingest videos,
train and optimize custom models on those videos,
use those custom models to program video analysis pipelines,
and then enable these custom pipelines
to be run efficiently.
In fact, we currently process over 400 million minutes
of video per month, which actually equates,
and we did the math, to about 761 years of video.
This approach has made a significant impact
in verticals like digital safety, where every online service
has its own content moderation policies,
and every piece of content being uploaded
needs to be moderated at scale.
But maybe they can't afford to pay millions of dollars
for that level of coverage.
It has also created new experiences in gaming
and Esports broadcasting
where every game has different analysis needs.
So, let's see a real example
of Watch For's processing capabilities.
The hub is the place to find the best eSport streams,
stats and news for many top Esports titles today.
Watch For's applied to tens of thousands of public streams
in real time to identify the most interesting content
related to that game title.
It can identify details like streamers
who have a high number of kills in Gears of War multiplayer,
or maybe someone
who's building a really cool structure in Minecraft.
This means it's easier for users to find content
from top streamers, but also makes it possible
to catch something you wouldn't normally see,
like an amateur streaming their best game ever.
If you're looking for specific moments,
you can also filter each game by certain modes,
maps and rolls.
What's happening in the background here?
Watch For's indexing live streams
from various public streaming providers in real time.
It's looking for streams
where there are a certain number of players left.
It looks at how many kills that player has,
what mode and so much more.
Each stream is processed by our custom analysis pipelines
and the metadata about what's happening in each game
is sent back to the hub in real time.
The hub then does further processing
on the relevant metadata
and the stream is then displayed to you.
The best part,
is that Watch For has a very low cost per stream
to enable these experiences.
Today, Watch For is used by several teams across Microsoft
for various video and live stream based scenarios.
It is still currently a tool set available for internal use,
but we hope to be able to share the Watch For platform
with you in the future.
In the meantime, you can learn more about the techniques
behind our work on the Microsoft Research Watch For page.
Thank you for your time.
- We have to actually rethink the entire AI stack.
- What is re-enforcement learning?
Let's look at an interactive demo.
All our code is now open source.
(bright upbeat music)