[MUSIC]
>> Super excited to invite
Abhishek to talk about Synapse Studio.
Hey, Abhishek. How is the going?
>> Hey, Sana. It's going good.
Thank you for having me here.
>> Thank you for joining us, and you're calling
us here from Shanghai, right?
>> Yes.
>> For those of who might not know,
can you just give us an overview of what
Synapse Studio is and why
Microsoft created this platform?
>> Absolutely. Azure Synapse Analytics is
a limitless analytic service that
brings together data integration,
enterprise data warehousing,
and big data analytics together.
Azure Synapse brings together the best of
SQL technologies used in enterprise data warehousing,
Spark technologies used for big data,
pipelines for data integration and ETL,
and deep integration with
other Azure services such as Power BI,
Cosmos DB and Azure ML.
>> What about devs who are familiar with Spark,
but they haven't necessarily used
Microsoft as a platform.
Why should they use this platform?
>> Sure. As I truly call this,
as a truly unified experience because
you get Apache Spark
or Azure Synapse deeply and
seamlessly integrates the Apache Spark engine,
the most popular open-source big data engine
used for data preparation,
data engineering, ETL, and
machine learning in Synapse Analytics.
That's one of the key advantages
that you get using Synapse.
With it, it brings ML models,
the Spark ML and gardens Azure ML integration
for Apache Spark is pretty rich, and with Spark 3.0,
which was announced at Build support,
it helps you build data lake and also Delta Lakes,
which is based on Linux Foundation.
It simplifies the resource model and that frees you
from having to worry about managing clusters.
Big data had its own challenges
in terms of managing the resources,
huge clusters,
and the amount that you end up paying for those.
All those things are taken care
of as part of
Synapse Analytics and the Spark offering in there.
I think one of the key important things as well,
there is the built-in support for document,
for Spark, allowing you to reuse your C# expertise,
which many of us coming from
the Microsoft Tech Stat
background would already have and
so you can leverage that skill set to build
spark applications as well using .NET code.
>> That's awesome to hear. This is apart from,
like you said, allows you to manage data,
whether it's in warehouses, data lakes,
but it also allows you to move and transform that data.
I think that's called the workflows.
Can you tell us more about that?
>> Absolutely. Let's talk about
Synapse pipelines to be specific.
That's what we want to talk about.
A few capabilities that you get as part of it is,
you can ingest data at scale using 98 data sources.
That's one of the ETL challenges,
is data integration so you get all of those.
It offers you good free ETL,
which is known as dataflows.
I'm going to demo that right now with that.
Then you can orchestrate notebooks,
stored procedures, SQL scripts,
and much more with that.
Here on the screen,
you can see I have
a dataflow and I start with a very visual design.
This is how Synapse Analytics workspace looks like.
Here I am in the integration hub and I
have a dataflow open right in front of me.
I'm going to go deeper into this,
enter a full mode so
that when our data engineers
when they're working on this flow,
you can actually see things.
I start off with my source.
I have data and I have data from all my orders.
I have client information and all those things.
I'm going to do a few joins
and show some transformations.
Transformation is as easy as adding
just a few clicks and then you can
choose what transformation you want.
You want to join this orders table
from another table using a common key.
You can do Unions, Lookups,
you have a bunch of transformations
and you don't have to
write any code across through this.
Each and every step, you can actually go
hit the data preview and actually see the data,
validate the data if the projections are right,
if the transformations are right which you're doing,
and then finally arrive into the last step.
I have multiple steps where I do aggregate,
I join this data across
different tables with the intention
to be able to finally get some insights out of my data.
I would probably want to understand how's the sales
going on and what kind of category or
occupation of customers,
how much are they ordering
and what's the average per order that's
happening and probably do
some kind of a ranking so that I can pass on this list to
business analysts who can actually make
use of this and even
suggest certain sales programs
which can help the sales better.
>> Wow. These Synapse pipelines seem really powerful.
When developers often work,
they use source control collaboration,
they have different environments for working.
Can you do something like that if you're
a business analyst or
a developer working on these data pipelines?
>> Yes, absolutely. That's one of the keys.
You can configure Git repository
in Synapse Studio as well,
which makes the collaboration much easier.
It has good versioning,
it brings in all the beauty of source control,
distributed development resulting in better productivity
of data engineers and
data scientists working on Synapse Studio.
It also has the continuous integration story around it,
which is the process of automating the build and testing
the code every time a team member
commits changes to version control.
Continuous deployment is a process to build, test,
configure, and deploy
multiple testing and staging environments.
All of those are possible for
data engineers using Synapse Analytics now.
In an Azure Synapse Analytics workspace,
continuous integration and delivery
moves all entities from one environment,
let's call it dev environment,
the one demo which we saw was a dev environment,
and I can move all of these codes when I have done
a local testing into my test and
then later qualify for production environments.
>> That's great. It really makes it possible for
teams of data engineers to work on these together.
Let's bring it back, talk to me about
the cost when managing a lot of data.
There's a lot of overhead with that.
What's the cost of this and how do people get started?
>> Synapse pipelines run in
a civilized fashion and you are built only for
the execution durations so you can
configure whatever amount of compute your workloads need.
For example, if your data is
growing in size or if your data
requires a lot of compute in terms of transformations,
you can choose the compute of your choice
and you will be billed based on the execution duration.
It's extremely cost effective.
In a few words,
I would just like to sum it up,
it's big data processing made simple and cost effective.
>> We love to hear that, big data
made cost effective and simple.
Thank you so much Abhishek for joining us.
>> Thank you, Sana.