>> Hello, this is Lara Lubbelke with Azure Friday,
and today I have Thomas Alex,
who is the Principal Program Manager
on our AI platform team.
He's going to show an interesting solution that we use
internally for our internal first-party services.
Things like, Office 365, and SharePoint Online,
and some other big programs that we have within
Microsoft that leverages HDInsight Kafka
and a number of other services.
We call the Siphon.
So, today, Thomas is going to walk us through Siphon,
and how we use Siphon,
and help you understand how you can use
these services as well to build out your applications.
So, Thomas, what would you like to show us?
>> Hey Lara, thank you for having me on.
I would just like to tell you
a little bit about Siphon today.
I'll take you through what Siphon is
and show a quick architecture diagram of what Siphon is,
take you through a couple of examples of
customer scenarios that we power using Siphon,
and then I'll talk about why we used
HDInsight Kafka to power Siphon.
>> Great.
>> Okay. So, a quick overview of what Siphon is.
So, Siphon is what we call a data bus that enables
high scale near real-time data flow
for various services to produce and consume.
So, it really supports three things: Easy publication,
discovery, and subscription of data in near real-time.
Second is it enables
streaming processing especially using
open source platforms like Storm or Spark streaming.
Thirdly, a conflict driven way to route data
from various sources to
important first party destinations within the company.
So, this is what Siphon is.
As you said, it's an internal tool.
We use it for many teams within the company
especially in Bing and Office 365.
>> Great. Well, let's see how this is put together.
>> So, this is a quick overview diagram of Siphon.
Let's take a quick look from left to right.
There's an SDK that data producers
use to produce data to send to the service.
The second piece over there is
a collector service that receives
all the data and
does a few other things like authorization,
and make sure the data is secure,
and persists it into the Kafka cluster.
So, that's the heart of
the pap service that Siphon is providing.
On the right, you have the consumption services.
You have basically two modes of consumption.
Either you have streaming platforms like
Spark or Storm that are used to consume the data,
and we also have a connector service that can
be configured to push data to various destinations.
All of this is sort of wrapped up in an API that
is used by both the Siphon's user interface,
as well as other tools that indicate with Siphon.
>> Okay. So, all of this is essentially Siphon?
>> Exactly. The whole thing
comprises Siphon and we leverage,
HDInsight Kafka at the heart of this service.
>> Okay. So, we're using
our own HDInsight service composing it with
multiple other services that we've
created or that we're leveraging other Azure services.
>> Other pieces on Azure services.
Well, like for example, we do
leverage service skybrick on Azure,
and we leverage Azure Container service
to build some of these other pieces,
but the core where the data has persisted
and that is used for messaging is HDI Kafka.
>> Great. All right. Let's see some more.
>> So, I want to quickly give a couple of examples
that illustrate how we use this within the company.
The first example is from Office 365 Security.
So, Office 365 has
their customers data sitting in Microsoft's cloud, right?
So, protecting that data is of paramount importance.
So, one of the ways that's done is by trying to detect
any intrusions or malicious activity
that's happening on these cloud machines.
The way it's done is,
signals are taken from these services.
Signal such as, somebody is attempting a log in,
or there were some multiple log
in attempts happened on a machine, and so on.
So, these signals are taken from the host,
injested through Siphon, and
processed using a streaming processing pipeline.
They use machine learning and various
techniques to raise alerts in near real-time,
so you can actually respond really fast if
there's an intrusion or some
malicious activity that happens.
This is actually a very high scale
from our data volume perspective.
Siphon supports over seven million events
per second for this particular scenario.
So, you can tell that's a lot of data and we
need high reliability and
availability to support those kind of scenarios.
Let me give you one more scenario.
This one is piloting SharePoint Online.
So, SharePoint is one of
the Office 365 services that our customers use.
Couple of areas that are powered by Siphon,
that is analytics and reporting,
and they're intelligent features.
When I say analytics I mean,
if a customer has
an IT admin who wants to understand
what's happening with their SharePoint service,
they get to report.
So, some of the site analytics reports
is powered by Siphon.
The second is an intelligent feature
driven off of user activity.
User activity might be
an end user of SharePoint Online checking in a file,
or making an edit to their presentation.
When this happens, signals are pushed through Siphon,
and you have an intelligence service
that consumes that and maybe provides a feature,
for example, a push notification.
Like, maybe your colleague edited
a file and you get a push notification.
We are ramping up
the Siphon use for the scale for Office 365,
and we are deploying across the globe
actually because SharePoint Online is a global service,
and so we need presence in
different regions across the globe.
>> So essentially, an administrator
who has set up their own SharePoint Online environment,
so, they're one of our customers.
I'm a customer of SharePoint Online
and I'm setting up a push notification.
Behind the scenes, it's actually wiring into
Siphon and Siphon is
driving that experience for me as the consumer.
>> Perfect. That's exactly how it's set up.
>> All right.
>> So, I want to explain
why we chose HDInsight to power Siphon.
In the industry, Kafka
is the standard for Bob Soubrette high scale.
So, when we looked at how are we going to deploy Kafka,
we looked at what are the options available,
and Apache Kafka HDInsight was a good choice for us.
Here are a few reasons. So, the HDInsight team
provides a managed service so
it's kind of deployment
patching maintenance and so on with high SLA.
So, that was a very
important service that we could leverage.
Another thing is, especially
when we think about Office 365 and so on,
we need to provide a lot of
compliance guarantees on how we manage the service.
HDInsight already has a lot
of security and compliance guarantees built in.
So, we can build on top of that and just add
the pieces that are required for Office 365.
The third important piece for us was the flexibility.
So, with different scenarios going on,
we need to customize the clusters that we build in
terms of the kind of compute and kind of storage we want,
and HDInsight gives that flexibility.
You can choose the VM type,
you can choose how much disks to
attach to a particular VM,
and we can also customize the software
that actually runs on each VM.
So, these were important reasons why
Siphon was able to use Apache Kafka HDInsight.
In addition, innovations like
the Managed Disks really give us
the cost efficiency that we needed.
There are other valuable features like
you may get monitoring out of the box,
it's available in regions across the globe so,
when you want to scale with
your customers going from one region to another,
it's as easy as creating
another HDI Kafka cluster in a different region.
>> Right. So, it's pretty easy to just get started,
get things set up and standardized,
and then where you need to have
customization with machine size
or you can have that flexibility.
>> Yeah. You can go with the defaults or you
can choose to customize as you need for your scenarios [inaudible].
>> Right, but you still get
all the benefits of the managed service.
>> Exactly.
>> Great.
>> So, I'll just summarize the scale that
we have with Siphon because I think it may
be interesting for some of the viewers to know this.
Overall, across all the clusters that we have
on Apache Kafka HDInsight,
we have over a trillion events per day.
We think that's consistent volume and is still growing.
So, we are happy that
HDI Kafka is helped us to
scale to this level and
we're looking to continue to grow.
Today, we have more than 30 clusters
running on HDI Kafka,
and these clusters together have more than 600 brokers.
A typical cluster is about 10 nodes,
and each node has 10 disks attached.
So, that's about 100 terabytes
of storage within a single cluster.
In our stress testing,
we found it can sustain
more than one gigabyte per second of ingress.
We don't run that hard in production so,
we have overall,
about four gigabytes per second and it's growing.
So, that's the kind of scale we're at.
>> Right. What are you looking for next?
>> So, the HDInsight Kafka team,
we work very closely with them.
They shared a couple of features that are
upcoming that I just want to talk about.
We're looking forward to leveraging these in
Siphon particularly getting to Kafka 1.
So, we're looking forward to the bug fixes
and improvements that come from Kafka 1.
The other feature that they are looking to release
is fine grained security control with Apache Ranger.
So, we think this will help us
especially in areas of
security and compliance guarantees.
So, we're looking forward to
looking at these features in future.
>> Very good. Well, thank you very much.
It was great to learn more about what you're doing
with Siphon and the incredible scale that
we have with this service that you've developed
for powering many of
our services that our customers are using.
>> Absolutely.
>> Also to learn how you can pull
together these different services that we provide
on Azure to create
your own application service
that power your enterprise applications.
So, it was really
interesting to see how you're using Kafka.
>> Thank you. We think this will be
very interesting for our Azure customers to
know how we within Microsoft are using these services to
build higher scale and
very large services that support all of Microsoft.
>> Absolutely. Thank you again.
>> Thank you Lara.
>> Today we've been learning all
about Siphon on Azure Friday.