DAVID HOPE: [MUSIC] Hi everybody,
welcome to
our joint talk between Confluent and Elastic,
Scale observability and secure data in motion.
Today we're going to be talking about why
Confluent and Elastic are better together,
some customer use cases where we're actually
using Confluent and Elastic
together in real customer environments.
I'm going to talk a little bit about
Elastic Coleman Schema for observability and
security and then you can see what
else there is at Microsoft Build.
So I'm just going to introduce myself.
I'm David Hope, Product Marketing Director, Elastic.
I actually also worked at Confluent or I have
a pretty good perspective on
both the Confluent and Elastic part of this.
But we have actually also
got Anand on the call from Confluent,
Anand you want to introduce yourself?
ANAND VENEGUPAL: Absolutely. Hey, thanks
a lot to Elastic,
David, and thanks to Microsoft.
My name is Anand, and I lead
the Cloud ISV Alliances for Data and
Analytics Partnerships in Confluent.
DAVID HOPE: Thanks,
Anand. Confluent and Elastic
are obviously better together but first
we're going to talk a little bit about
data that Anand do you want to take it away?
ANAND VENEGUPAL: Thanks a lot, David,
and it's a pleasure to be here with all of you again.
Like David you said,
data is the foundation of all of this.
Let's talk a little bit about what's happening around us.
One of the most recent big accomplishments,
for example in the area of space and science is
the successful testing of the largest rocket ever
built that SpaceX is going to take to Mars.
I don't mean to be a SpaceX commercial here,
but the big focus on that first test,
even though the news media made
it look like, oh, it went and exploded,
it went beyond its first stage of what the objective was.
The data was the most important part
of what they got out of it so that they can
actually continue to build
the next stages of this whole journey
of the human race pretty much being
a two planet race as opposed to one-planet raised.
This is very, very big and the big focus I
want to bring here is the data and how
important and extracting data from everything we
do and the business processes
as it is happening in real time.
How many people are clicking, what happened to it?
What are the different stages of the consumer journey?
What are the different aspects
of the life cycle of what have people buy?
How's your network running right now?
Can we predict an outage?
All of this is data,
and a lot of this is data in real time.
The combination for example of Confluent and
Elastic makes this whole thing extremely powerful.
You bring the data in real time
from different sources to a platform,
for example like Elastic,
wherever you are observing it, you're searching it,
you're visualizing some of
the key insights so
that you can take immediate action and change.
We offer two things with that to customer's.
One is the real-time customized expedience as
personalized experiences
and efficient data-driven operations in the back-end.
Without a really great efficient back-end,
you really can't provide those
front and experiences to customers.
We, as consumers,
are really becoming very picky
choosy and raising our expectations all the time.
We want businesses to react to us in real time.
We want to sit
your e-commerce website anticipate
our needs and give us relevant products.
Really know and get foremost
in real time or what is happening around us.
For example the weather reports
or anything that's changing around us.
For businesses to be competitive,
you need these two things to be happening in parallel.
One is very, very efficient,
real-time data-driven operations in
the back-end and offering your customers
personalized experiences in the front that are
relevant, fresh, recent, etc.
That's how this partnership and
this combination of real-time data streaming platform
like Confluent and Elastic
as a search and observability platform
is going to be extremely helpful.
On the Microsoft Cloud, which is
probably the best Cloud for enterprise customers,
this is all going to work really well.
For what customers is this relevant?
Pretty much across industry verticals so it's a banking,
mortgage, taxi, grocery.
If you look at every aspect of our life right now,
it is rich with these experiences
where we are seeing highly interactive,
highly real-time, much quicker responses from
the system that is
enabling us to take actions much faster.
This really surprising for me
the first experience of having
a car got filling a form and
then waiting weeks to get approved versus
the most recent experience of
literally getting a credit card of
more than probably $10,000
credit limit within a few minutes.
That's a business process that's occurring like a miracle
to a current consumer
compared to the previous generation.
Similarly, banks are becoming digitized.
We all are used to the Ubers and Instacart.
It's a completely digitized,
real-time transformed world that we're living in.
There's still a lot of
distance to go in terms of making age-old,
archaic batch oriented business processes
moved from where they
are to much more real time in nature.
The technology behind all of this is Apache Kafka,
which was invented in LinkedIn
originally and now Confluent is the company that
offers enterprise class Apache Kafka
with a connector with it's Cloud-native,
it's extremely high-performance, way better
than running your open source Kafka cluster on your own.
The most important part is all of
these partners that work with
us to make this whole thing real,
the whole aspect of moving data
from the sources to the destination virtual time.
Being able to take some powerful platform like Elastic at
the end of that data flow and being able to
examine the data in real time and take action on it.
I think it's critical.
I mean, imagine cybersecurity as a use case.
Imagine if you were not able to detect
cybersecurity events fast enough.
The difference between detecting an event
even before actually damage occurs or
a breach occurs versus after a day after it happens could
be tens of millions of dollars
in terms of penalties and
data breaches and things like that.
That's the power and importance of this technology.
Comprehend is happy to be right there
at the crux of all of this.
The difference between Confluent
and open-source Apache Kafka
is that we make this enterprise class
would provide you the security.
We provide you the enterprise class interconnectors all
around Kafka for partners to connect in,
for interfaces to the sources, to the destinations.
Also much more enhanced functionality in terms
of stream processing on the stream.
So you might want to build out your transformations,
your anomaly detection, your analytics,
etc, on the stream itself,
and all of that and being
a Cloud-native platform with
extreme scalability and performance and
elasticity is available on Cloud.
You can see this architecture here in front of you,
which is APIs and sources
of events coming through into the class that's
the publish-subscribe model and there's
the central part that gives you
the ability to process data
and then the ability to deliver data to the n
destinations all in real time is a continuous flow.
Elastic would be on the right side of
this as a consumer of that real time flow
so that an analyst origin could be looking at
it in real time and analyzing it and taking actions.
That's what we call the
centralized nervous system of the underprice.
To make this all real and how
we work to gether to bring Confluent and
Elastic and handover to
David and tell you some real stories
of how are we bringing this together.
DAVID HOPE: Thanks very much And.
Hopefully that last stake
really go together like peanut butter and jelly,
complementing each other in ways that empower
businesses to leverage come
from and data streaming effectively.
And the last thing, office three main solution areas
there are especially relevant to
comfort when users search,
observability and a security.
Now, while most people are
familiar with elastic search capabilities,
it also serves as
a powerful observability and security tool.
Many organizations also rely
on Elastic for logging purposes.
With the staggering volume of logs generated,
it's crucial to have a scalable solution.
Both Confluent and Cloud plays
a vital role in alleviating back-pressure,
allowing businesses to stream and analyze logs in
real-time and ultimately benefiting
our shared customer base.
Elastic versatility doesn't just stop there.
We also can use it for traces,
application performance monitoring and metrics.
In that case, it's the perfect tool
for monitoring both the comfortable platform,
comfort, Cloud, and data in motion applications.
With such a wide range of applications,
the Elastic Stack is
a valuable assets in numerous use cases.
Today, we will discuss how to utilize
Confluent as a pipeline for open telemetry data
and we'll explore three key use cases where
both Confluent and elastic
provide essential support to customers.
Finally, we'll talk about
Elastic recent contribution to the CNCF and
open telemetry projects to come up with
a standard schema for telemetry and security data.
OpenTelemetry is
an open-source observability framework,
and it's revolutionized the way we collect and
transmit telemetry data such
as traces, metrics, and logs.
By integrating it with Confluent Cloud,
a fully managed event streaming platform,
we can seamlessly route the data
to Elastic for analysis and visualization,
which is where we are strongest.
Confluent Cloud enables us to collect and
stream the open OpenTelemetry data in real time
offering a scalable and reliable event
streaming capability.
By using Confluent Cloud is that bridge,
we can decouple our telemetry data from
our observability solution providing
flexibility and resilience in our architecture.
Let's talk a little bit about
some customer stories where we've actually
seen this in real life out there in the wild.
Now Elastic and Confluent power one of
the top software vendors' observability,
security information event management,
and MLOps solutions,
it transforms to enterprise
Cloud data management capabilities.
This customer, a leader in the field,
helps their customers transform
their most critical assets and
realize business value from using
their end-to-end management platform with over
85 of the Fortune 100 clients relying on them.
It's essential to have
a robust observability solution
in place to support their customers.
With Elastic Stack and Confluent,
this customer collects data across
four different Cloud providers in
20 regions with hundreds of applications,
three Kubernetes clusters,
and 8,000 virtual machines.
This generates an enormous amount of data,
37 terabytes a day.
It includes logs, metrics, traces,
and security logs, all
streams through the Confluent platform and into Elastic.
To ingest the data they use Kafka,
Elasticsearch, Kibana, and Logstash,
and it forms the backbone of their solution,
providing powerful metrics and
insights on application performance.
Elastic helps them in
various ways such as improving search and discovery,
providing real-time insights for incidence,
acting as a single plane of glass,
optimizing costs and enhancing security.
Machine learning in Elastic plays
a vital role in predicting application performance,
detecting anomalies, and full-costing data volume.
Moreover, Elastic's APM solution
enables quick identification of
bottlenecks in those microservices and
simplifies root cause analysis.
Next up we have a financial services customer.
Here, we dive into the magical world of
log management where Logstash, Confluent Cloud,
and Elastic Cloud join forces to create
an unbeatable trio that will
transform the way
the financial services handle their data.
First of all, let's talk about reliability.
How heroic trio ensures that no log is left
behind as they bravely
decouple ingestion from processing,
making sure that even in the event of adversity,
system failures, the things that keep you up at night,
your precious data remains safe and sound.
Now imagine a world where logs and
messages can be prioritized.
Our fearless friends make
this possible enabling
heterogeneous service-level agreements
for different business units.
It's like having a VIP section
for your most important logs.
As the volume of logging data grows, worry not,
our magnificent trio scales effortlessly,
handling the most overwhelming logging demands.
They're like superheroes ready to take
on any challenge thrown at them.
Last but not least, our heroic trio of Logstash,
Confluent Cloud, Elastic Cloud bring
credible flexibility to the table.
They're the ultimate log management chameleons
supporting diverse log sources and destinations,
adapting to any environment they find themselves in.
Finally, we examine a customer who
runs a large betting operation.
They use Confluent Cloud and Elastic and
Logstash to combat debit card fraud, money laundering,
and bonus fraud where fraudsters sign up
using stolen identities to
claim lucrative welcome offers.
Application owners across their business can leverage
Confluent Cloud and the
Elastic Stack to effectively manage logs,
monitor events, produce alerts,
and ensure seamless data processing and analysis.
How do they do this?
Well, first, logs are generated by application owners
across the organization and sent to Confluent Cloud.
This centralized approach to
log management enables businesses to
effectively monitor and analyze
data from a wide range of sources.
Next, Logstash comes into play and pulls logs from
Confluent Cloud and in
some cases enriches them with additional data.
Once processed, Logstash ships
the logs to Elasticsearch for storage and indexing.
Elasticsearch also has watchers.
As logs are stored and indexed within Elasticsearch,
the watcher feature continuously queries log indices
to identify potential issues, patterns, or anomalies.
If any of these are identified,
the watcher generates alerts
to notify relevant stakeholders,
ensuring that fraud can be stopped promptly.
Now, I just want to talk a little bit
about our recent contribution
to the CNCF and OpenTelemetry projects.
We recently donated the Elastic Common Schema
for unified observability and security schema.
In the bad old days,
we used to have this situation
where messages from anywhere,
any telemetry source,
logs, traces, metrics, anyway really,
would have a different name for the same piece of data,
which creates a nightmare because you end up with
this proliferation of all these different data
that actually means the same thing.
Too often we see that it
causes organizations to struggle.
They don't understand why something
happened or did something happen,
and the root cause of all of this
is very difficult to determine.
This is why Elastic drove to develop
the Elastic Common Schema to
unify data across an organization from logs,
metrics, traces, and security events.
Now, we live in a world where, essentially,
we can get rid of all of the different data
that essentially means the same thing
and refer to it as one thing,
which makes it much easier to pull all these things
together and get the context
we need across all of this data.
In the context of Confluent,
we also could integrate the schema into
the Schema Registry to make sure that any own ramping
of observability data conforms to this valid schema so we
don't end up in the situation where we have
this proliferation of field names.
As you can see here,
the Elastic Common Schema plus
the OpenTelemetry Semantic Conventions
equals the new OTel Common Schema,
which has logs and security schemas donated from Elastic.
Thank you very much, everybody.
Stop by the DevSecOps meet-up room to learn more
about how Confluent and
Elastic can empower your business.