SPEAKER 1: What happens
with objects at rest?
They stay at rest.
But what about objects in motion?
They keep moving.
Action, reaction.
In a world where everything is in
motion with your data at rest,
your customer experiences your ambitions,
your business itself will stay at rest.
Sit that data in motion and every impulse.
Every action triggers
instant reaction across every experience,
every app, every system across your entire business.
Tap into a continuous stream of
data in ever flowing nervous system that
unlocks those real-time insights that are
only insightful because they're real time.
The world doesn't wait.
The world moves,
and your data needs to move too.
JACOB BOGIE: Hello and welcome to power
real-time data streams from
anywhere to Azure Cosmos DB with Confluent.
I'm Jacob Bogie and I'm
a Cloud Platform Solution Architect here at Confluent.
I'm thankful that you chose to spend some time
with me to learn more about how you
can enable real-time data
streaming pipelines using Confluent Cloud
and Azure Cosmos DB.
There is unprecedented growth in the volumes
of data being generated by businesses today.
It is now essential for enterprises to be able to
contextualize and harness the value of their data and,
along with that, gain
real-time insights into how their business is performing.
This enables data teams to
quickly make operational decisions
in response to market conditions
from anywhere at anytime.
Modern data architectures are
requiring real-time data support.
This is creating a need for
a new data strategy that includes
data streaming platforms and
transactional data stores working in unison.
Traditional self-managed databases are
difficult to manage at scale.
You need to consider all of the costs of running,
operating, and managing the lifecycle
of a large-scale database offering.
Integrating with other systems is complex.
Creating and maintaining
these integrated services becomes
a new learning experience anytime you
add a new pipeline of data into this pattern.
By migrating to a Cloud-hosted system,
you can increase agility through
native integrations with modern features
that you don't have to maintain.
Here are five situations to look out
for and plan to address
as you journey to delivering
a democratized data as a product offering.
Number 1 is batch ETL and ELT.
Batch-based pipelines use batch ingestion,
batch processing,
and batch delivery, resulting in low fidelity,
inconsistent, and stale data.
Number 2, centralized data teams.
Bottlenecks from a centralized domain agnostic data team
hinders self-service data access
and can slow the pace of innovation.
Number 3, immature governance and observability.
Patchworks of point-to-point pipelines
have high overhead and lack observability,
data lineage, schema, and data error management.
Number 4, infrastructure-heavy data processing.
Traditional pipelines require intensive,
unpredictable computing and storage,
emit high data volumes
and an increasing variety of workloads.
Number 5, the monolithic design.
Rigid black box pipelines are
difficult to change or port across environments,
so engineers often choose to just
add one more for each new workload,
thereby increasing pipeline sprawl and technical death.
Why Confluent and Azure?
Well, our partnership enables
modern data platforms to connect
disparate data sources and
power the use of Cloud-native data services.
Confluent and Microsoft announced
a strategic partnership in January of 2021.
Confluent Cloud is a Microsoft-preferred solution
that has been validated to generate business impact,
technology transformation,
and cross-organizational improvements for its customers.
Confluent is much more than Kafka.
Confluent is a Cloud-native and complete platform
for data in motion and event streaming.
Confluent is available everywhere.
It's on the Cloud of your choice,
especially Microsoft Azure.
It is offered as
a fully-managed zero operations service,
and you can also run Confluent in your own data server.
Confluent also includes ksqlDB
for stream processing and streaming analytics,
a full suite of managed connectors to
other Cloud services and
third-party systems like databases,
enterprise security and data governance,
infinite storage for your data streams,
and much, much more.
Confluent breaks down data silos
to help teams quickly access and transform
datasets for real-time use cases
and decision-making no matter where the data lives.
We can enhance your data in real time,
transforming data in flight using ksqlDB,
and we break down silos using
our connectors to pull in
data from any data sources you have.
All of this on a fully managed set of
services that can help you focus on building
things instead of running them and extending
your data's reach to new business scenarios.
With Confluent, you can integrate and stream data from
on-premise data sources or
the Cloud services of your choice.
In this case, we are using Microsoft Azure Cosmos DB.
It's going to act as
our modern multimodal data store in the Cloud.
Let's take a closer look at why we
emphasize what Confluent can
do for you when using it as a solution to connect,
process, and govern data.
Connecting existing data sources to
new data sources is easy with Confluent,
with over 120 pre-built connectors,
many of them fully managed in Confluent Cloud.
You can save time and effort by not having
to create your own data integrations.
Our sink and source connectors connect
the places where data lives and where data needs to go.
They do this all while
offering single message transforms,
so you can further customize your data
during ingress or egress of your cluster.
In Confluent Cloud, you can process data
quickly and easily with ksqlDB.
ksqlDB supports aggregations, joins,
windowed-based queries, order handling,
and even exactly one's semantics.
All of this occurs in
computational and storage resources
that are separate from your Confluent clusters.
ksqlDB is also highly available and fault-tolerant.
Confluent's stream governance suite is
the only fully managed governance suite
for Apache Kafka and data in motion.
Stream governance establishes trust in the data streams,
moving throughout your Cloud environments and delivers
an easy self-service experience
for more teams to discover,
understand, and put streaming data pipelines to work.
The three main components
in the stream governance offering
are stream quality where you can deliver trusted,
high-quality data streams to the business and
maintain data integrity as your services evolve;
stream catalog where you can increase collaboration and
productivity with self-service data discovery
that allows teams to classify,
organize, and find the data streams that they need;
and finally, stream lineage,
where you can understand
complex data relationships and uncover
more insights with interactive end-to-end maps
of the data streams in your environments.
You can see in this example architecture that we
have connected on-prem data
sources using open-source Kafka
and our supported on-premise offering Confluent platform.
We can link these clusters with
Confluent Cloud and enable
a streaming data mesh between on-premise
existing data silos which had been broken down with
Confluent platform and Confluent Cloud being
connected and create a new Cloud data mesh pattern.
Once leveraging the power of Confluent Cloud,
we open up the opportunity to stream data to
Azure Cosmos DB and create
a geo-replicated multimodal data architecture that
enables our use of
other Azure services to meet the needs of our business.
We have also integrated Azure functions to provide us
with true scale to
zero serverless application functionality,
which extends our data mesh to being an active data mesh.
As our time here today comes to an end,
remember that the technology components are
just one piece of a recipe for success.
This recipe also includes the organization of
the teams that are delivering
your new self-service offering and,
of course, the governance model of the datasets
that you've created with this new data mesh.
Let's wrap up with a few assets for you to take away
and help you and your teams begin your journey.
You can scan the QR codes on the screen to get access
to a demo of our technology in action,
an e-book detailing the path
to streaming data pipelines,
a solution brief that focuses on
Confluent and Azure Cosmos DB.
You can visit our Cloud database solution web page.
Finally, you can get started
for free using Confluent Cloud on
Azure by visiting the Azure marketplace
and starting your free environment today.
Thank you for your interest in
Confluent and I hope this has been helpful to you.
SPEAKER 1: What happens
with objects at rest?
They stay at rest.
But what about objects in motion?
They keep moving.
Action, reaction.
In a world where everything is in
motion with your data at rest,
your customer experiences your ambitions,
your business itself will stay at rest.
Set that data in motion and every impulse.
Every action triggers
instant reaction across every experience,
every app, every system across your entire business.
Tap into a continuous stream of
data in ever flowing nervous system that
unlocks those real-time insights that
are insightful because they're real time.
The world doesn't wait.
The world moves,
and your data needs to move too.