>> Learn how to use Citus to scale out Postgres,
this week on Data Exposed.
[MUSIC]
Hi,
I'm Anna Hoffman,
and welcome to this episode of Data Exposed.
Today I'm joined by Claire,
a Program Manager and leader on the Citus team at Microsoft.
Hi Claire, how's it going?
>> Good, happy to be here.
>> We are happy to have you here and
Claire before we get into the topic for today,
I would love if you could give us a brief intro on what you do.
>> What do I do?
I do all sorts of things.
I'm on the Citus open-source team.
Citus is an extension of Postgres and
I do a lot of work to help people discover Citus,
learn about what it is, how it works,
why they should care,
how Citus can help them.
The engineers that I work with are spread actually
all over the globe in five or six different countries.
So in my work at Microsoft,
I end up with early morning meetings and late-night meetings,
and I'm based in California.
Yeah, I've been doing this at
Microsoft for about two and a half years and
I was at Citus before
the acquisition for a couple of years before that.
>> Awesome, cool. We're excited to have you on the show and I
think you're alluding to what my first question is,
which is really like, can you tell us a little
bit more about what is Citus?
>> Absolutely. Citus is an extension to Postgres.
Postgres, I think many of
your viewers know it's an open-source database,
hugely popular,
used by people and applications all around the globe.
Well, I can bring up a slide too if we want to display.
>> Sure.
>> Let me click ahead to this guy.
I'm on the Postgres team at Microsoft obviously.
But Citus is an open-source extension to Postgres
and what Citus does is it
transforms Postgres into a distributed database.
It's super useful for applications that need
scale that goes beyond a single database server,
they need more memory, they need more compute.
Maybe those applications are really growing like gangbusters,
not just more data, but more users,
more activity, more features, things like that.
So Citus enables you to continue to use Postgres,
but as a distributed cluster,
especially for applications that have already
started using Postgres from the very beginning.
Citus can be an incredibly transformative way to grow that up.
Yes. That make sense?
>> Yes, no, that means a lot of sense.
One question you talked about
distributed and I think a lot of that has to do with sharding.
Can you tell us a little bit about
how sharding plays into Citus and
also what obligations you see a lot of people using this for?
>> Yeah, I can definitely do that.
Before I do, I just want to pop this up on the screen
just to drive home the point that
Citus is an open-source extension,
and this is just a screenshot of
the GitHub repo where you can find all the discussions,
and the activity, and the things that
the engineers are doing on the project.
Citus is also available on Azure as well as managed service,
which is incredibly useful to people who
run their applications in the Cloud on Azure.
But you wanted to know about what use cases or what applications?
Repeat your question, please.
>> Yeah, sure. I was asking,
what kind of applications do you see sharding Postgres with Citus?
>> Okay. We see multi-tenant SaaS applications because those are
good examples of apps that tend to grow really fast and
need that scale at some point along in their evolution,
and SaaS apps typically like to run in the Cloud.
They want to outsource the database operations,
let somebody else take care of security and backups,
and all of the data protection things that need to happen.
SaaS apps is probably number one.
Number two, I would say our real-time analytics dashboards.
Think of customer-facing dashboards
where there's a lot of concurrency.
There are a lot of users hitting that dashboard at
the same time doing interactive queries.
They don't want some pre-baked dashboard that
was created a couple of days
ago on data that's a couple of days old.
They want to do interactive queries on real-time data,
data that some people call fresh,
that's just been ingested and can already be
queried in order to answer the questions.
We see a lot of those applications and they ultimately need
the parallelization that you can get
by using a distributed database, that is Postgres.
Then we'll sometimes see IoT applications where
those IoT applications also
need the benefits of relational database,
they need updates, they need joins,
things like that and then there's
just super high-throughput transactional
applications that just need
the scale that you can get out of distributed Postgres.
>> Awesome. Let's double-click a little bit into the scale side.
How large can a Citus cluster get?
>> That's actually a really good question.
There's a blog post that I wrote called
Architecting Petabyte Scale Analytics
by Scaling Up Postgres with Citus,
and there's a customer actually at Microsoft,
that was a customer before the acquisition,
before I worked for Microsoft.
He uses Citus and Postgres and he runs it on
Azure for a database that is multiple petabytes.
He's not the only one. We have
other customers I could talk about where they
have these petabyte-scale platforms.
When you have a distributed database,
when you have 60 nodes that are all serving your application,
obviously you can grow really, really big.
In his case, I'm looking at
the actual details in the footnotes right now.
His application needs to support about 10 million queries a
day with 20,000 different diagnostic metrics.
He has hundreds of
concurrent users who are all
hitting that dashboard at the same time,
and I believe it's about 60 nodes.
I'll look that up while we're talking here. Things can grow big.
But I think one of the exciting things also
with Citus is that earlier this year,
you and I talked on Data Exposed live back in May,
and there was a Citus 10 release that we issued that had
some pretty exciting new capabilities and one of them was
this ability to shard Postgres on a single node.
Even when you're just getting started,
even when you know you're going to need to scale
later this year or sometime next year,
you can actually start to use Citus to
shard and to parallelize your workload
on a single node and that makes it much easier to
transition to a distributed database later when you need to.
>> Cool. Awesome. That's very cool.
Claire, this has been really useful because
I've gotten to learn and our viewers have gotten to
learn a little bit about the Citus use cases
and some of the great benefits.
I just have to ask, is there a situation where you would say,
maybe Citus isn't the best solution for an application?
>> Yes. If your application isn't growing,
if it doesn't need to scale,
then you might not ever need to use Citus,
if you don't expect to have
demanding performance requirements at any point in time.
There are some applications that work really well
on a single node and that app's not growing,
and they might not ever need to use Citus.
But what we find when we talk to customers
is even if they're not growing now, they plan to,
they expect to and they want
the improved performance that you can get from
the parallelization and from sharding.
>> Awesome. No, I think that's a great answer,
and I think like everything,
the answer is, it depends on your scenario.
This has been really useful.
As we start to wrap up the episode,
I wanted to ask if you had any final tips, tricks,
advice that you wanted to share with our viewers about Citus.
>> I have a couple of things,
I want to share some of the ways you can get started,
so let me see if I can share anything here.
We put together a bunch of material.
People learn in different ways.
So whether you like to read documentation or watch demos or
roll up your sleeves and try
the tutorials and download the open source packages,
we have this wonderful Getting Started page
that my team has created.
I have a short form URL showing on screen right now.
That'll show you how to get there and there's
just all sorts of useful information there.
The Citus docs are pretty awesome too.
A lot of our users have given those a
plus one and said thank you to the team that writes them.
That's one thing I wanted to point out.
Also, if you are interested
in extensions to Postgres and how those work,
how those use the Postgres extension APIs and you want
to understand why we built Citus the way we built it,
why it's architected the way it is,
this is a super useful video that just got published that
was created by the technical lead of the open-source project.
I put another short-form URL on the screen too,
if you want to go watch how Citus distributes Postgres.
It's based on a SIGMOD paper that actually Marco and
team delivered just a couple of months ago as
well. That's the second thing.
The third thing I wanted to flag is
that earlier this year, in Citus 10,
we added new columnar storage capabilities to Citus,
which is pretty powerful.
If you have 10 minutes, actually
could be a little bit more than that and,
you want to watch a demo of what it is
and what it does, Jeff Davis,
who's the architect of that feature he
created this video and I put a short URL there too.
Those are the three things I wanted to share.
>> Awesome. Thanks so much Claire,
this has been really informational.
I know some users are going to want to go deeper in this.
I want them all to know that we've put
links to all the things Claire talked about in the description,
so you're definitely going to want to check those out.
Claire, again, just thanks so
much for joining us. This was really useful.
I think our viewers found it useful.
To our viewers, if you liked this video,
go ahead and give it a like,
subscribe to our channel,
leave us a comment, and let us know what you think of
Citus and we hope to see you next time on Data Exposed.
[MUSIC]