[upbeat music]
>>Hi, welcome back to THWACKcamp,
my name is Thomas LaRock, and today we're gonna
talk about a developer's perspective
on distributed Software as a Service database monitoring.
Now you might know me more as a DBA.
I was a developer in a former life,
but in order to help me with the developer part
today, I've invited my old friend
Karlo, Karlo say hello.
>>Hello everybody Karlo Zatylny, distinguished engineer
at SolarWinds.
Happy to talk about
some cloud native database development with everybody.
>>That's great because I like how you said cloud native.
I think cloud native is a misused term.
I often hear people just say
oh, we're gonna look at this database.
It's a cloud native database.
And to me, cloud native, isn't about a database,
it's about an application or a system.
Do you find some type of misinformation around there?
The use of the word cloud native?
>>Absolutely, cloud native means different things
to different people.
And so what we really need to do is just
clarify what that means.
And it's, about taking things into consideration
like cost, multitenancy, security,
How do we address those
while having a publicly exploded service?
To me, that's cloud native and building things in a way that
satisfies all those business needs that we need to satisfy
>>I know, I think that's an important point,
and I usually tell people
when you think something cloud native
think of something that is also globally distributed.
So just because it's a database
doesn't make it a cloud native database.
Any database could be the back end
for a cloud native system.
That's what I try to remind people.
Now as a developer, I think
I think some of the tools that you use might be saying more
of the application performance monitoring side
and we have one of those that we call AppOptics.
Is that tool that you find yourself using a lot of?
>>Yeah absolutely, that's super valuable for us
to use as developers, so why don't we take a look at that
and jump right in.
>>Awesome.
>>So as we look at AppOptics Tom,
we can see that this is just
a classic API call into one of our environments,
and it has a different view into the different layers
that are going on,
but importantly, this is one of my query
or one of my API calls that has a few different queries
executed the during the call,
and so when I'm looking at the traces and AppOptics
I often will find that a lot of the time is being spent
in specific queries as they're getting executed
through the system
and AppOptics gives me great information
about the actual execution time
of the query as it's going from my trace into the database,
but what I need to know as a developer is
if this query time is okay, or could it be better?
There's lots of things that as developers
we need to learn as we start to write code
in a cloud native way,
because we won't always understand
how the software is going to be used by our customers.
So as it is running in production,
we need to take the time to evaluate is 771 milliseconds
really an okay execution time?
And if it's not,
what are the actions I'm going to
take to remediate that?
Will I need to change the architecture?
Or database design?
Or will I need to consider
another database store altogether?
You mentioned S3 earlier
and it's definitely a great option for us to store data in
with MySQL being another one.
But as we go cloud native,
we have the opportunity to look
at some other highly specialized databases
that allow us to do certain types of data storage
and more efficient ways,
whether that's numeric time series
or logs or search index documents.
Whatever data we're dealing with,
there's a lot more options given to us
as cloud native developers
that does require us to work closely
with our DPA's and DevOps to take full advantage
of a more diverse architecture
because we don't need to necessarily
have to think about
how our customer is going to have to deal
with these on-premises.
We can grab more efficiency
by using the variety that comes with being cloud native.
This gives us the opportunity to take a look at query time
and determine is this MySQL even the right target
for this type of data?
If it is, great,
if not, do I need to look at Cassandra?
Or ClickHouse? To store something
in a more efficient way for my data.
And that's part of the power of AppOptics
that gives us the links into DPM
where DPM can start giving me insights
into the analysis that I need to say,
is my target actually the right place for my data?
Am I executing at the level that this needs to be?
>>So, if I understand this,
so this is one API call here that
actually has three specific SQL queries, is that correct?
>>That is correct, yeah.
>>Yeah, and so we can see,
so even as a dev right now
like I said, if somebody had joined the company
and they're working on this app
and they knew nothing about it, this lets them see
oh, they might have only known one query.
And they're like, well, where are these other two?
And you could look at these
and say, well, do I really need to run all three of these?
Or could I run just one query
and have that data available to satisfy whatever is needed
for the API call right?
So this is the efficiency part, right?
So this whole API call took about 1.7 seconds.
It looks like 1.69 was just on these three queries correct?
>>Exactly.
>>Yeah, so maybe
that that costs money, that's money in cloud right?
So maybe we could think about some consolidation
or just tuning the queries themselves.
>>Right, precisely, and that's where we need to start
diving into the database side of things
to evaluate the performance not only for the customers,
but to your point the cost is also a big concern.
Am I writing and designing the software
in a way that our company can also make money
and meet the business needs
of both our customers and ourselves.
>>So for cloud native, I think I have this right.
As I shift from the traditional legacy,
everything in my datacenter
or at least under the dev's desk,
that's where everything all the data is, right?
Let me think, it's throughput, latency, concurrency,
and errors, is that right?
>>Yeah, that's >>These are
my four golden metrics.
>>Yeah, latency, traffic, errors, and saturation.
Those are the golden metrics that we've talked
about before in different sessions.
That's the magic.
Are we getting the best performance
from a latency perspective?
And if not, are we getting higher latency
because of the saturation being too high?
Are we getting a lot of requests to a specific place
that we weren't expecting?
Or are we seeing a lot of errors?
And are we getting the data throughput that we need?
Or are we sending too much useless data or not enough data?
So yes, it's hitting all those golden signals
and super important for us as developers to know
that we are writing the most efficient code
to give the experience that the end user is expecting.
>>Well, I happen to know
where those metrics are inside of DPM.
Do you wanna take a look?
>>Yeah that'd be great.
Let's take a look.
>>Awesome, so Karlo,
I've got the summary page up for DPM right here,
and as you can see there, we've got
about 20 oops,
we've got about 20 hosts connected.
So we're looking at basically an aggregate summary
across all of these hosts.
That's not really helpful
but we need to know is as a developer
you're gonna come to me and say,
hey, I need some information, I'm running this app,
and I may or may not have an awareness
of all the database servers that are running
or grouped or anything like that.
So with DPM, once you've registered machines
you can also group them and then you can see down here.
So I'll just look and say, all right,
I know it's the API server,
and I know this is, I have a primary, a primary.
I know there's a secondary here api4.
I'm just gonna assume that this group
of servers, this helps give me that
high level information to understand
you know, what's really happening here or
at least the architecture involved.
And you may or may not say to me,
oh, it's the problem is over the API server
or MongoDB or a WebApp server,
but with this inventory pain, at least I have an idea
of the machines that might be involved.
So if I come back to the summary page
in real quick, if I just say,
if I type in here api4.
We'll say, this is the MongoDB server I wanna look at,
so now I can filter, and now I'm gonna get just a view of
just that host.
And remember I talked about those golden metrics.
So right here,
right around here, database health.
So latency, throughput, concurrency, error rate.
These are the golden signals.
And I can see if I hover over it,
I can see exactly at 11:41 when the polling happened,
I can see the correlate in metrics
for the resource health as well.
So now you've got your CPU, your network
your disc, your memory,
these are all things familiar to any database administrator
whether you are new to the field, or you've been doing it
for a long time, and you're old
in your experience like I am.
The metrics, the resource health metrics,
those are the things every database has.
And you can also include locking and blocking as well.
So top queries by count, oops,
I keep having trouble to scroll.
So I, down here I also get top queries by count
top queries by total time,
and that's where you as the developer might say,
oh, I know which one that is.
It's this particular query that we're trying to look at.
So we had an idea of the queries by count total time.
That gives me good information.
If I'm ever wondering what this
summary page might wanna tell me,
DPM has a wealth of knowledge-based articles
hidden inside of it to give you an understanding
of what you're really looking at.
So I'm looking at the summary page
for this particular environment.
I can also see the api4 actually covers
there's four different nodes
or hosts involved in just filtering on that api4
So if I remove that filter real quick
and we just say, Karlo might come to me and say
you know what, it's the top query by total the time,
and I really need you to help me figure out what we can do
to optimize that.
So now it brings up that query
and Karlo was kind enough to tell me
this is what he wants information by or about.
I can look right now, I'm looking
at a metric of total time
and I can see when it's wrong and I get an idea
of some of the metrics involved at that particular moment.
I can tag things, which is really important.
Karlo would tell you anything related to DevOps
tagging is extremely important.
So if you had, were deploying changes,
you could then come in and say all right,
I wanna tag this particular query
in whatever way.
Annotate it however you need to.
Now, what I see here are an idea of the host breakdown.
So webapp, app2, app3
if I'm supposed to have
an understanding of how much time is spent
like if I'm using some load balancing,
I might wanna think
that these should be a little more equal, maybe, maybe not.
I see performance, I see average latency,
total time, the count so 8,540 executions here
in this particular timeframe,
which is a one-hour timeframe.
And additional metrics, how many rows,
how many of the queries were slow
and the 99th percentile of latency.
So this is giving me information at a pretty good level
for me to give back to a developer to understand
the real effect that this particular query is having.
So, seeing this information, Karlo
what are some of the things that you might start considering
as a developer in order to optimize this particular query?
>>Well, as I look at this data,
there are a lot of things I can do
with the software to optimize performance,
looking at the host breakdown.
Oftentimes we're going to look
at sharding the data across different servers
to optimize performance.
So seeing that there's a misbalance might tell me
that I did my shards across my servers
in an incorrect way, so I might reconsider that
'cause I shard maybe to be multi-tenant,
maybe I'm doing my shards based off of a certain customers
go to different shards within my system,
and maybe that's not the right way to do it.
So as I look at the data,
I might need to reconsider how I'm distributing my data
across my different servers.
So it's very powerful and insightful for me to come in
and see those percentages,
so I can say maybe my approach isn't the best
or it's looking good
and I'm willing to keep my shards that way.
That's because with cloud native, what I'm really
looking to do is have a nice distributed system
in a way that makes sense for my customers and for my data.
So being able to see that distribution really tells me
if my design is succeeding at its intended purpose
or should I be looking to shard based
off of other parameters
like time, geo, or what those variables are
that I should be paying attention to
in fine-tuning the sharding.
The other thing that I'm looking at here is
the latency, total time, and count,
if I see that something is having
a strange amount of latency, I might need to look
and say, what's going on at those specific times
is my data partitioned in a way that makes sense?
Again in a multi-tenant environment,
what we might be looking
for is partitioning our data off of user ID
because I don't want other users being able
to see each other's data from different organizations
or maybe I am partitioning based off of time.
Am I storing data based off of time of day and day of week
when those specific events happened?
So if there are specific problematic areas
we might need to look at.
If the database is doing some summarization routines
or performing backups during those high latency times,
maybe I need to adjust my partitioning
in a way that yields query effectiveness
and gives the performance we need consistently over time.
As I look at the query counts
I think, do I need to look more into caching?
Caching can be such a valuable tool inside of the software
because what we're able to do is take
some data that is written very infrequently
but read very frequently
and put that into a cache
so that we're not consistently hitting the database
and putting some unnecessary load
and unnecessary traffic
and unnecessary latency back and forth.
So that's a really key performance counter for me
to determine if the software is behaving
in a way that I'm expecting.
As a developer, you try to make guesses and estimates,
but then when it goes out into the real world
you might notice, oh no, they're hitting this all the time.
And this is a really popular feature,
and I wasn't expecting that.
Or maybe the software isn't being used
in the way that I was hoping for the software to be used.
And this data gives me the opportunity to make a change.
So if I see that something is being read a lot, written
not very often, there's the opportunity to use a cache.
>>Now, in my experience, cache is one
of those ways that you can
really reduce costs in the cloud right?
'Cause that's, >>Yeah absolutely
>>That's one of those optimizations,
like if you can identify ways that you just have to read
the data once or get it into cache,
and then just read from cache.
You get great performance and you reduce the bill
'cause you're not moving that data
back and forth all the time.
>>Exactly, and cloud native is such a great opportunity
to use services provided by your cloud platform
or use one of the many open source projects available.
And these software components are available in a way that
make it so you don't have to be an expert
to get the cost benefit you gain
from using such a technology.
Also, there is no large amount of overhead
to your engineering, as you can take that
off the shelf service and immediately see the benefit.
>>So I'm gonna finish up here.
I was gonna point out the health.
One of the things I like as a DBA
is this best practices right?
So I can come right here.
And we've talked about how
and we'll just imagine that a lot of these
are supposed say are
the back end for cloud native applications.
And I can come in here and say, all right, this Postgres
I have nine recommendations, 45 occurrences.
This gives me information, it says, all right,
what's my cache? Is this something actionable that I can do?
It should be configured with a value of 50 and 75%.
This is a great way for me, 'cause I may have experienced
with PostgreS, but I may not have as much with MySQL
or MongoDB, but the information is right here for me.
And I can come in here and I can get the details.
I have an idea as a DBA.
I know there's only four bottlenecks memory CPU,
network, and disc, right?
So coming in here, this gives me a nice summary
and I can be, have a positive effect
on performance for all these different platforms.
I don't need to wait to get,
10 or 15 years experience in just one particular area
a tool like DPM helps identify these best practices.
I understand what it means to go in
and rebuild an index or update stats, things of that nature.
So I really like this and how it was summarizes me
and I can filter and say show me the critical ones.
They're all right here.
As a matter of fact, these are configuration.
Yeah, I can just get an idea of what's really happening
in my environment on a regular basis.
I can see events, I can see faults as well inside the
MySQL or PostgreS engine.
Nice high-level view for me,
in addition to looking at that summary view
or exploring and profiling and comparing queries over time.
What DPM really does is it gives
me that ability to be effective
especially in the era where I think a lot
of apps are really starting to become, like I said
globally distributed, cloud native in a lot of cases
when people are thinking about going to cloud these days
they do think I'll just lift and shift.
And then they realize that probably wasn't the right choice.
Now they're starting to think,
we're gonna build something new
and we're gonna build it from the ground up,
and we're gonna do it with a brand new architecture.
And a lot of the traditional database monitoring tools,
they aren't really configured to help monitor those type
of application architectures, DPM is.
I wanna thank you Karlo, for your time today.
I appreciate your developer's perspective.
This was a lot of fun.
We should chat more often.
>>Yeah definitely.
>>All right for THWACKcamp,
I'm Thomas Larock.
>>And I'm Karlo, thanks for watching
and enjoy the rest of the THWACKcamp.