EVREN ERYUREK: Hello, everyone.
I'm Evren Eryurek.
Welcome to Streaming Analytics on Google Cloud.
When you look at IDC, they predict by 2025,
more than a quarter of the data in the global datasphere
will be real time in nature.
Now, you may wonder, why is this important?
It is important because window of opportunity
in the real-time analysis diminishes and goes away really
fast.
You want to be able to actually respond to your customer
needs, their asks, be able to do prediction or maybe detect
some problem and really respond to it really fast.
While each and every one of this data
becomes valuable for retrospective analysis,
time to respond in real time is very limited.
So you want to have a platform that you
can achieve both of these, real time
as well as a place where you can do retrospective analysis.
And for that, you have Stream Analytics on Google Cloud.
And this is the platform--
many of you out there are already
providing various solutions in your industries.
You have Pub/Sub on one end, the Google native service
that entire Alphabet company takes advantage of.
We made it available through GCP for you
to take advantage of as well.
And you're doing a great job in there.
That is your ingestion and distribution of data, reliably,
globally.
And you have DataFlow for all your data
processing, whether it's streaming or batch.
Now you can do that.
And as you know, in DataFlow, it's
the same set of code for both streaming and batch.
And then you have the AI Platform, BigQuery, BigTable,
Cloud SQL.
All of them are there for your machine
learning needs, for your AI needs and data warehousing
needs, to do more analysis for your businesses.
Here are some examples and great success stories.
Home Depot-- when they started their journey
on the digital transformation, when
they wanted to establish this e-commerce platform
for themselves, they picked Pub/Sub, Dataflow, and BigQuery
to do this thing.
And this allowed them to really increase
new revenue opportunities and reduce their churn.
When you look at Palo Alto Networks,
they are building their next-generation firewall
using Dataflow as an embedded data processing system.
AirAsia uses the streaming analytics platform
to provide refined pricing for their customers.
This allowed them to increase their revenue
and improve customer satisfaction and customer
experience.
ULTA Beauty has established their e-commerce platform where
they can actually provide real-time offers
to their users, and suggestions based on their shopping
practices and search practices and so forth,
again using this real-time analytics
platform for their shoppers.
Now, in this platform, in your own organizations,
you have some heroes out there.
Your data analysts, your engineers,
your data scientists, and your operators
are out there making this work for your systems,
for your customers.
And throughout my conversation today with you,
I'll be highlighting what we are doing
to make their lives easier, more effective,
and also bringing some cost saving opportunities
for all of you out there.
Here's a snapshot of what you're going to hear
for the rest of my talk.
You will hear Dataflow SQL.
We're actually really happy.
All the feedback that we've gotten from you
on the early releases--
you're loving this thing.
It made it much easier for your analysts and engineers.
You're really enjoying the AI Platform integration
with Dataflow.
And Jupyter Notebooks was one of those things
that data scientists really, really asked for it.
And here you go.
You have an integrated and seamless experience
from many of our services on Google Cloud,
including Dataflow.
And operators have asked for some dashboard,
some real-time analysis capabilities,
and a lot more in here.
We're going to go into it.
Ready?
Let's go.
Let's start with the ingestion first.
I mentioned, Pub/Sub is one of those services
that we across Alphabet leverage.
And we made it available.
It's a globally scalable, fantastic technology--
really, really good.
We did hear loud and clear from all of you
that you wanted message ordering and dead letter topics.
And thank you for your patience.
Now, they are going to be available for you.
We also heard some cost saving opportunities
where you asked for a service like Pub/Sub Lite.
I'm going to go into it a little bit more.
And it gives you cost-optimized event ingestion and data
messaging platform for your zonal needs.
A very global nature may not be necessary for you.
So we created in here--
our objective was to make it so compelling,
even for your do-it-yourself Kafka solutions
that you may have, it is going to be a higher cost than what
we can provide to you.
And of course, through our Confluent Cloud partnership,
we have a fully managed Kafka service
for all your on-prem to cloud migration needs.
And CDC was one of those capabilities
that you've been asking for.
And I'm going to give you a couple examples of what
is coming up for you.
Here is a Pub/Sub Lite.
So this is possibly the lowest cost
to start possible messaging and event messaging system
that you can find in here.
It's ideal for predictable workloads.
It's zonal, so that you can actually
figure out where to do it.
It's really easy to configure and already seamlessly
integrated with Dataflow, rest of Google Cloud Data Analytics
Platform, and GCP services.
So give it a try if you haven't used it.
We believe it will be much easier for you to really take
advantage of Pub/Sub Lite than do-it-yourself Kafka-like
environment that you may have.
If you have a lot of Kafka workloads,
and you are migrating to Google Cloud, you are using--
or you should be using--
Confluent Cloud.
We have seamlessly integrated with our partners.
They are integrated just like Pub/Sub with Dataflow
and all our other GCP services for you
to achieve your Kafka workloads as you
are migrating towards GCP.
And CDC-- this was your ask.
And here's two options for you that we're actually
providing you.
And for your MySQL and Postgres use cases,
we're building a solution with Debezium.
It's available as an open source, self-managed template.
As a fully-managed template solution from ourselves,
we're bringing you-- through our Datastream,
you will have a CDC capability for Oracle Databases.
Through Datastream, through Dataflow, through BigQuery,
you'll be able to provide this CDC capability.
This has been one of those asks.
And we're happy to show you what's coming up for you.
For all data engineers and some pipeline development--
that's key.
I highlighted the Dataflow SQL.
It's for you to really take advantage of SQL.
It's the same dialect that we have
with our BigQuery and Spanner.
You'll be able to do kind of joins
within the Pub/Sub streams, and BigQuery tables,
and GCS files, and so forth.
You will do some schema supports using the Visual Editor.
It's right there for you.
Your early responses were very, very positive.
If you haven't started using it, I
encourage you to really take a look at it.
You'll be able to build your real-time dashboards based
on BigQuery.
Do your transforms.
Start there-- very easy.
Go at it if you haven't started using it.
Data scientists have been asking for this Jupyter Notebook
integration with Dataflow, so that you can actually
do your own interactive development
prototyping in the environment that you're so used to it.
Now, Dataflow is integrated into Jupyter Notebooks.
Go ahead and do it.
You have the interactive runners there, all the support
that you need for your AI Platform on Google Cloud.
It's the same experience.
So this was something you've been asking for.
I'm really happy to announce that it is in your hands.
Now, this is a good one--
Flex templates.
As you know, we provide 25 different templates
through our Dataflow UI and CLI for you to take
advantage of as a template.
One of the asks that many of you had was,
hey, I built these Dataflow pipelines into my system.
But I want to turn this into a template,
so the rest of my organization can use it as well,
within my own team, within my own organization.
Now, you can do that.
This was something that you asked for.
It's a much better system.
And you're used to it.
You created it.
It's all yours along with the 25 that we provided.
Quick Recommendations-- this is a really interesting
capability.
So it actually provides you guidance
for your data analysts, data engineers, data scientists.
And it achieves some cost saving along
the way where you can actually get some recommendations.
Hey, do it this way and do it that way.
Maybe clean your Datastore databases or extra entities.
Find different ways to restore it
and backup and restore Spanner, what have you.
It provides you, through these recommendations,
ways to really improve it.
What is it that you can do?
And it is all in the fingertips of yours.
Performances scale.
This is always a paramount for us.
This is always top of mind.
We always figure out a way to make it faster, easier,
and scale even bigger and broader.
Here's one-- FlexRS.
If you haven't heard, this is a fantastic
cost-saving opportunity for you.
You could save up to 40% of your cost in some of your batch
workers.
If you have a job that you run daily or weekly,
you don't have to worry about changing a single line of code.
All you have to do is turn it on.
Add one pipeline option.
That's it.
But remember, this needs to be one of those jobs
that you run daily or weekly, because it
might take a few hours to find the most cost
optimum environment for it to run.
So that's why it's says in here, it may delay up to six hours.
Here's what we have done for the Beam community.
The Dataflow Runner v2 is available.
It has a lot more capabilities in there,
from more I/Os, including Kafka I/O and so forth,
to parity across SDKs, including the state and timer
support in Python.
It's available right now for you to test with your Python
streaming pipelines.
Go at it.
And try it.
You will find some cost-saving opportunities there, all of you
data engineers out there.
And thank you for really taking advantage of the Beam.
And our operators, for you--
you've been asking this dashboard.
Now, all your important pipeline metrics
in a single time-correlated dashboard is ready.
It's available for you.
You'll be able to go deep dive into it.
Understand how we are making some of these autoscaling
decisions.
That was somewhat of a mystery for all of you.
And you've been asking for it.
So we're providing some of this visibility and actually
some cost reductions as well.
So you can optimize your own pipelines,
latencies, and throughputs as you're going at it.
And these are all in your fingertips.
And keep asking.
Keep pushing us in this area.
Managed pipelines-- this is another one
that you've been asking.
Now, you'll be able to understand resource allocations
over multiple job executions that you may have.
And this actually enables you to really
give you better freshness on installers and so forth.
Drill down on your pipelines.
Maybe figure out some problems that you
may be seeing at the end, and fix it, and optimize it.
It's much easier for you to do it.
We're going to continue to invest in here.
Ease of use is paramount for us.
We really, really want to make it easier for you.
Now, real-time AI-- as an AI student,
I'm always excited about what we're doing in here.
So we've been working with our partners across the ecosystem
and trying to figure out what is it that we can do,
and how is it that we can actually create some patterns
and create some kind of easy-to-start templates
and sample codes and so forth by integrating Dataflow and Cloud
AI Platform for pattern recognition,
anomaly detections, and predictions and forecasting
and so forth, for every industry you can think of,
from media to retail to health care to fintech.
You name it.
There's some great capabilities out there.
You can use structured and unstructured data
for your pattern recognition.
You can use it with your videos.
You can use it with images and text and so forth.
Now, here's an example from Telus.
A great team in Canada, they are one of the largest and fastest
growing telecom companies.
And they have many, many events coming in.
And they wanted to actually use Dataflow
to detect some anomalies in their network.
So we're looking at 250,000 events per second
that they process.
And they figure out, are there any anomalies
that we need to be really looking at it
and detect it in a timely manner and respond to it?
This is one of those use cases where you really
need to detect fast and respond fast.
And Telus team has done a fantastic job taking advantage
of this platform, these capabilities
that we provided with Dataflow.
And there is a lot more to do there.
You can do it in every industry.
If you visit our website, you will see a lot more in there.
And the codes are ready for you to take
advantage of from GitHub.
Let's wrap it up together.
We are the leader in streaming analytics
when you look at Forrester's Streaming Wave.
There are a lot of things that makes
us the leader in this area.
There are five criteria that we've
been rated the highest in here.
Many of you are already taking advantage
of these capabilities.
And we invite all of you to come.
Take a look at it.
Visit.
Ping us.
Chat with us.
There is a way we can help you with all your needs in here.
Here is a list of you out there doing wonderful things
everywhere in the world, from retail to transportation,
from health care to gaming to media to fintech.
All of you have built great solutions on these technologies
that I just spoke about.
Thank you for everything that you're doing.
Thank you for all your use cases,
and challenging us, and working with us.
We're looking forward to really hearing more of your feedback.
We're here to really work with you.
Here's a wrap up for you.
We talked about what we've done for data analysts, data
engineers, data scientists, and operators.
There's a lot more out there if you go and visit
our streaming analytics website in Google Cloud.
You will find many blogs, many links to the code repositories,
and so forth, to really make it easy.
But if you're still having problems, ping us.
Ping me.
Find me at wherever you can.
We'll get you going right away.
Thank you for your time.
We really appreciate it and hope you had a wonderful time.