Now we've all heard of the hyper scalars
the Facebooks, the Googles of the world,
but did you know, they have a trick up their sleeve,
they actually profile every single
line of code in their data centers
and it allows them to answer the question,
what is our most expensive
line of code
and then they can go and attack that
line of code and make their services
more efficient and faster for their
customers.
And what Elastic is going to do for you
for our customers is we're democratizing
that capability so that you can
profile everything all the time.
All code, all systems.
and how do we do that?
Well, first of all, we're using this new technology
called eBPF, which is something
that's built into newer Linux kernels.
And what it allows us to do is it allows
us to provide profiling with zero instrumentation
which means that the administrative overhead
of deploying this solution is significantly less
And you can start to get insights
into what your most expensive lines
of code are immediately
with no code changes and no app restarts.
Next, we'll give you whole system visibility
from the kernel
through user space, into high
level code, like java across
all your cloud environments.
We do this with extremely low
overhead so that you can put this on your
most critical applications and
start to get unprecedented visibility
into which lines of code are
causing you the most pain.
We'll help resolve and improve
your issues with faster
meantime to discovery,
we'll identify and analyze your
most expensive lines of code across
your entire fleet, across all
of your data centers
and we'll do that with all
of your applications. It doesn't matter whether
they're written in Ruby, C++
PHP or Java.
Now we've got a demo for you that shows
you how you can get all of these
benefits in your environment
and we're gonna go through three scenarios here
The first one is
pretty typical. I've seen this myself
in production in a number
of places. Why is my
service initially slow when
I restart my JVM or why does
the JVM take so long to start up?
So that's the first thing we're gonna have a look at.
The second thing we're gonna have a look at is where
is my most expensive line of code
And we're gonna look at elasticsearch and see if
there's anything in elasticsearch that
we could potentially go and improve.
And finally we've got an issue
that we've seen where when we scale up
a particular service is suffering,
can we find out what lines of code are causing
that problem?
We certainly can.
So let's dive into the demo,
So this is Elastic Universal Profiling
And the first page you'll
see here is a view
of all the stack traces taken from
all the containers within our environment.
And you'll see here various container
names like beats for example,
which is a familiar Elastic service
if you are also familiar with Elastic.
Now these are all the containers and it shows
how much CPU time each container
is taking up.
But we actually want to get a little bit more granular
than that in order to solve our JVM issue
We really want to see what
individual threads are doing which
is a little bit more granular than individual containers.
So let's dive into the threads view over here.
So here we have the threads view and
we can see which threads are taking
up the most CPU
And actually one stands out straight
away. This one that's in pink here
definitely looks like it's taking up
more CPU
let's have a look at what that thread is doing
Ah, it's a C2 compiler thread
Now, if you're familiar with java, you know
that what this thread does is it compiles
classes, particularly,
it's gonna to be active when we restart our JVM
So we can see here that indeed
compiling classes is taking up
a lot of time when we restart
our JVM. Now there's a lot of interesting
ways that we could potentially solve this problem.
And one of those ways is we could tweak
the JVM startup options to
make it take less time compiling classes
on startup or we could even
compile our java classes to native
code ahead of time. Using JVM
That's the one I like most. So I'm gonna
go ahead and do that and I'll be saving
a significant amount of CPU
If you look here across
all these instances when I go ahead
and do that
So it looks like we've we've made a good
dent in one particular issue.
So let's move on to the next issue.
How do we find our most expensive line of code
in the elasticsearch service?
So let's scroll down here
and we have elastic searches.
So I'm just gonna drill down into elasticsearch there
and that brings us to traces. It brings
us to traces that have been taken
of our elasticsearch service
Now, if we go down here we can see the top
100 traces that have been taken
and we can start to see the
stack trace for each one
of those traces. Now this is
great, but I really want a holistic view.
I want to see what are the top
100 methods
across all of these stack traces that are slow
within my environment. So I'm just gonna
head over to the functions tab, which is over here
and have a look at exactly that.
So here's the top 100
methods that are taking up the most
CPU within the elasticsearch service
And there's one here that stands out to me this
one at number three. This hashing
function I know from my own personal
experience that that is something that we
could replace very very quickly
with a different hashing function and
probably save ourselves a significant
amount of CPU
So this is one of our most expensive
pieces of code and we've
very quickly determined that we
can make it much more efficient,
Which will mean that it will be much better
for our customers. It will be a faster piece
of code. It'll run more efficiently.
It will use up less C02.
And so finally for our last
particular problem that we're trying
trying to solve here, we we've got this issue
where when we scale up a particular
piece of code doesn't like it when
we're trying to scale up to
more pods in our environment.
So we wanna go and check out that particular issue
So I'm gonna go over here.
and we'll see the containers view again
and we can see here that
in fact actually there are
some containers that don't seem to like it when
we scale up because this time period here
is one of the time periods where we're doing
a scale up event and then after that
it kind of goes back to normal. But during a scale
up event there's a few
containers here that do not seem
to like being scaled up and there's one
in particular here called vmedia.
And I'm gonna just drill down into this
vmedia container and try
to understand a little bit more about what's
happening there and this time I'm gonna
use the flame graph. So I can
get a view of all the methods
and all the functions that are running within
that container that are taking up
time on the CPU. So let's just
go over there and have a little look at that
Ok, so there's some methods here.
One that actually stands out immediately
to me is this token info method,
we can actually see that that's taking up quite
a lot of the CPU
Now I wonder why that is. This looks like
an authentication operation that's happening.
So I think probably what's happening here
is that when we scale up
authentication seems to be suffering
a little bit, it's not, it's not coping
too well with that scale up event.
The one thing I wanna do is I wanna compare
This flame graph to another flame
graph that was taken previously
when we weren't scaling up so that I can
see that yes, this piece of code is
suffering when we do a scale up
event. So we can compare a flamegraph
that was taken
previous to our scale up event
and a flamegraph that was taken
during our scale up event.
And I can do that by clicking on this differential
flamegraph tab here.
Now, what the differential flamegraph tab
will do is it will take that comparison
and where it sees that a particular
method call has gotten worse,
you know? So during the scale up event
it will color it in red.
And actually we can see here that
yes, this particular method,
this token in the method is
performing worse during
a scale up event because it is actually
slightly red as you can see there.
So this has been great.
We've actually managed to solve three
different issues using the
Elastic Universal Profiling tool.
And I'm very happy now that I have
a lot of things I can do to go
and save my customers time,
money, and make the services more efficient.
Thanks very much for listening everybody.