(upbeat music)
>> From San Mateo, it's theCUBE,
covering Scalyr Innovation Day,
brought to you by Scalyr.
>> Hello and welcome to the special Cube Innovation Day here
in Silicon Valley in San Mateo, California
at Scalyr's Headquarters.
I'm John Furrier, host of theCUBE.
John Hart's the Tech Lead Back End Engineering
here at Scalyr.
Thanks for having us.
>> Thanks for having me John.
>> So what's the secret sauce at Scalyr?
You guys have unique differentiate as we have covered
with some of your peers and the founders
are all talking about it.
But, you guys have a unique secret sauce.
Take a minute to explain that.
>> I think, yeah, it's a few different things.
First of all, you've got just the design level,
which is we don't use keyword indexes.
So that's a big one right there off the top.
On top of that,
you've got a couple of different implementation paths.
We've got our own custom written data store.
So we're able to really control all the way
down to the bytes on disk,
how we lay things out,
optimize for speed.
We have a novel kind of scatter-gather approach
for fanning out a query,
to make sure we can get all of our nodes involved
as quickly as possible.
Then, finally, and this is just kind of being smart,
which is we have a time series database
for repetitive queries
and that's on demand.
You don't have to do anything, but we're going to speed up
your queries in the background if we know it's a good idea.
>> Talk about the time series.
I think that's interesting because that comes to play.
We hear about real time a lot.
We talk a lot about in cyber security
that time series has been beneficial.
Where does time series fit for you guys in here?
>> That's a good question.
I think one of the big differences with Scalyr
versus other uses of time series database is
with Scalyr you're outputting your logs,
there's all kinds of information in your logs.
Some of that might be a good thing to put
in a time series database,
but I think with a lot of other products,
you would have to decide that ahead of time.
Like, hey, let's get this metric into the database.
With Scalyr, the moment you have anything
in your logs that you might want to put into a time series
you just start querying it.
You put in a dashboard (snaps) you've got a time series.
So we're going to back propagate that
for everything you've already given us.
So all of those queries are fast from there on out.
>> So it's built in from the beginning.
>> Exactly, and you don't have to do anything.
It's just on demand.
>> So keywords been what other people have been used
for years.
That's been standard for these log management software
packages and indexes.
Indexes can slow things down.
We've got a tutorial on that.
Why is those two areas, haven't been innovated in awhile?
When people just haven't figured it out,
you guys have first?
What's the differentiation for you guys?
Why'd you guys get there?
>> I think the main reason is that log data is
just fundamentally different than most other things
that you might use a database for.
There's a couple of different reasons for that.
So with log data, you're not in control of it.
You can't design it.
You know, an index is great if you're making
a relational database.
You've got control of your columns.
You know what you're going to join on.
You know what you want to index.
Nobody designs their logs like they design their
database tables.
It's just a bunch of stuff.
It's from systems you don't control.
It's changing all the time.
So just the number of distinct fields
that you would have to index is really, really high.
So if your system depends on indexing for good performance,
you're going to have to make a lot of indexes.
And indexes, of course, they're right amplifying.
If you've got one gigabyte of raw data,
then you've got to put five or six hundred indexes
on top of it.
You're going to have five or ten gigabytes of raw plus
index data.
That means you got to do a lot more IO,
and at the end of the day,
how much you have to read from disk,
determines how fast your query's going to be.
>> So, in essence indexes creates a lot of overhead.
You shouldn't even need to do because of the nature
of log files.
>> Because the nature of log data, it's overhead
that doesn't serve log data very well, yeah.
>> And what about the log data that's changing?
Cause one of the things we're seeing, Internet of Things,
more connected devices,
imagine the Teslas that are going to be connecting in,
with all their data. >> Right
>> All this stuff, cameras.
You've got a huge amount of new kind of data.
Up, down, status.
This is going to be a tsunami of new types of log data.
>> Yeah, and none of it are you going to have a ton
of control over.
Right, it's going to be changing a ton.
Maybe you've got 20 different versions of devices
out there that are all sending you different versions
of logs.
You've got to be able to handle all of it.
So you want a system that is adaptive
to your needs as they come up,
as opposed to something you have to plan out
with indexes ahead of time.
>> So if someone asks you, say you guys say you're faster.
Why?
Is that true?
Is the statement you're faster than others,
and if so why?
>> It is true.
(laughs)
And that really comes down to the secret sauce.
The brute force, the key to brute force,
and I think we've talked about this a little bit today,
is you got to bring a lot of force,
as quickly as you possibly can.
And we do that.
We've got a lot of custom code.
We're not using off-the-shelf components.
We're trying to get that time quick as we can.
So I think our median performance is still better
than 100 milliseconds.
That might be for a query that's talking to
two or three hundred machines, or maybe even more.
All of which, to get,
maybe it's going to scan a terabyte of data.
All of that is going to come back within 100 milliseconds.
It's extremely fast.
>> Talk about why log data is different
from other data types, for folks that are
in these cloud native environments.
Their time is precious.
They are looking at a lot of different data.
How is log data different?
>> I think the fact that it's dynamic in terms
of what's coming out is something new.
It changes so rapidly.
The other really big thing too
is the way you query it changes from day to day.
Most of the time you're going to your logs,
you're trying to troubleshoot a problem.
Today's problems are different than yesterdays problems.
So every time you go in,
you're using it in a different way.
So it has to be very fast.
It has to be exploratory.
And that's one of the big things about Scalyr's speed.
Is it enables this really exploratory.
You can kind of move through the data quickly,
as opposed to making a query, getting a cup of coffee,
waiting for the query, and then deciding what you're going
to do next.
I'm kind of dating myself here,
but it's like the first time you ever used Google.
You're like, "Whoa, how did that happen?"
That's what it's like the first time you use Scalyr.
>> And you guys have a unique architecture,
we talked about that.
You guys have certain speeds.
But it's not just the query speed.
It's the time it takes to do the query.
So you factor in a much bigger
perspective than if someone has to build a query
and then takes 15 minutes.
>> Right.
>> Game's over.
>> Yeah, and instead you're just clicking on things.
We're trying to make it very easy for you to move
from oh here's an alert.
Well here are the log files that caused that alert.
Oh, what's the thread stack for that particular lock.
Oh, I can go and look at everything else that happened
in that thread.
That's five or 10 seconds of Scalyr tops.
>> You guys have unique engineering culture,
that targets engineers,
products built by engineers,
for engineers. >> Yep.
>> Great story.
And it's real, and you guys building it everyday.
What is the engineer threshold of pain when it comes
to locked data?
Have you seen any anecdotal, I mean,
'cause engineers that are in this space,
they need access to it.
There's SLAs now tied to it.
People are sharing data.
There's all kind of new ways, reasons why you need
to have the Scalyr solution.
But what's the pain point for most people
to tolerate an inferior solution?
>> Well for me, I actually have an answer for this.
Right, because before I was Scalyr employee,
I was a Scalyr customer and before I was a Scalyr customer,
I was a Splunk customer.
I used Splunk for about five years before I think Scalyr
even necessarily existed and I was really happy
with it because I needed it.
Right?
I had my own company.
We were generating tons of logs.
My support guys needed to use those logs.
And, prior to using something like a Splunk,
I was SSHing it to servers to check the log files,
which is of course, not scalable.
So I was really happy with the product as an idea existed,
but it just kept gnawing at us.
You know, every time we would query,
sometimes it would be fast,
sometimes it would be really slow.
Sometimes the results would be down
because an indexing server was down.
It was just.
>> You mean the Splunk solution?
>> Yeah, the Splunk solution.
Yeah, it was just extremely painful.
So I read, actually, one of the blog posts
written by Steve Newman and thought, that's a great idea.
That is how you should attack this problem.
New indexes.
Brute forces.
All the flexibility you get from that.
I loved it and then I forgot about it for like six months.
(laughs)
Because I was busy, right.
But then six months later I was really frustrated again
with Splunk again being really, really slow,
and I thought, what was the name of that company again?
I looked them up.
I installed it.
And within, certainly within a day,
I was blown away by the performance.
Within a week, I had uninstalled Scalyr, excuse me, Splunk,
from every single one of my servers and switched
to Scalyr instead.
>> And you're happy with that?
Does it work for you?
Came to join the company?
>> Yeah, exactly.
In kind of conversations with the support team here,
I was one of their early customers to use Windows,
so I had a lot of questions, they had questions for me,
how did I get it working,
it wasn't a supported platform.
And all of my emails were responded to
by two guys named Steve.
So I figured that was probably the support team.
Pretty funny they've got a support team of two people,
both named Steve.
And then at one point, in one email, Steve Newman
said to me,
"You may have realized there's only two of us here."
And that's when I kind of went,
"Oh wait, so there's two people total."
And two guys I assumed in a basement.
They weren't in a basement,
but I assumed they were in a basement.
They had software that was way better for my needs
than Splunk, which at the time was worth probably
eight, ten billion dollars.
It's a public company.
Thousands of engineers.
So that's when I thought, "Huh. When I get a chance,
"Maybe I should go work with these guys."
>> You know it's interesting.
Maybe create a new category, brute force as a service.
>> Yeah.
>> This is what they're doing.
They're bringing in the right tool at the right time.
>> Yep.
>> For the right problem, for speed,
and to solve the problem, no?
>> Yeah.
>> They care how it gets done.
>> Get as much data as you can
and get that answer back as quickly as you can.
>> So this is the big challenge.
Final question for you is obviously,
you know, a lot of people we talked to in the DevOps world
they're really fickle.
On one hand, they'll try anything.
If they like it, they'll stay with it.
But if they don't, you'll know about it.
Where's the value point for people to start thinking
about Scalyr.
Is it ingest to value, ingesting is one part,
that's kind of a trial.
Where's the value immediately come in?
Where do you see, what's the first sign of light value,
once the ingestion happens.
>> So part of it is this,
it's a very short period of time from the ingestion
to the time you're querying on it
is very, very short.
So you got a real time view
of what's happening on your servers
not a five minutes ago view.
That by itself can pay for it right there.
If you're a DevOps person and you've got some alarm pinging.
If that alarm is from 10 minutes ago,
that means your customers are already annoyed.
If you're going to have to wait another 10 minutes
just to even see what's happening,
you've got a really big problem, right.
So being able to have the alarm,
and you know that's triggering on something
that happened a second or two ago,
and then immediately being able to dive in
with no interruption to your work flow,
no reason not to dive in,
that's a pretty big one right there.
>> So pretty immediate impact.
>> Yeah.
>> So okay, for people that don't know Scalyr,
what should they know about Scalyr as a company
from a value proposition
as a former customer now, key employee in the back end,
and engineering.
What is the key things they should know about?
>> So speed, we keep talking about it, right?
We have a really really good cost basis.
Because we're not making those indexes,
we don't have to store as much data.
It's just generally cheaper for it to run.
Right, so we actually have a really good cost point.
And we get you from the alerts.
You don't have to decide stuff ahead of time.
You can do it all on the fly, ad hoc,
we get you from the alerts, to your answers
as quickly as you possibly can.
That's pretty good.
>> Every culture has its own unique kind of feature.
What's Scalyr's culture here?
I mean Intel was Moore's law,
Cadence was Moore's law.
What's the culture here, at Scalyr like?
>> That's a good question.
I guess I would say I'm just tremendously proud
to be working with these engineers.
Right?
We're all here because we want to get better
and we want to work on really, really hard problems
writing our own code,
not just running and kind of patching together
open source systems that already exist.
We want to be doing something cutting edge.
So that's I would say the biggest one.
>> And big problem's behind that,
you've got AI right around the corner.
Applying AI is going to be a natural extension.
>> Yeah, 'cause we got the data.
And can deal with the data.
>> Ciao, thanks for the insight.
Appreciate it. >> Thank you.
Good talking to you. >> John Furrier here.
Innovation Day with theCUBE here in Silicon Valley
in San Mateo, at Scalyr's headquarters.
I'm John Furrier.
Thanks for watching.
(upbeat music)