>> Announcer: From around the globe,
it's theCUBE.
With coverage of KubeCon
and CloudNativeCon Europe 2021-Virtual.
Brought to you by Red Hat
the Cloud Native Computing Foundation
and ecosystem partners.
>> Hello, and welcome back to theCUBE's coverage
of KubeCon '21, CloudNativeCon '21-virtual.
I'm John Furrier, host of theCUBE.
We're here with a great segment with an entrepreneur
also, the creator and maintainer of Fluent Bit,
Eduardo Silva, who is now the founder Calyptia
was a startup going to commercialize
and have an enterprise grade Fluentd and Fluent Bit.
Eduardo, great to have you on thanks for coming on theCUBE.
>> John thanks for having me here.
So I'm pretty happy
to share the news about the career
and whatever you want to talk about.
>> Exciting trends happening
with CNCF Kube Con Cloud Native Con.
Cloud Native a lot of data,
a lot of management, a lot of logging,
a lot of observability, lot of end user contributions
and enterprise adoption so let's get into it
first by giving us a quick update on Fluent Bit
anything upcoming to highlight?
>> Yeah, well absolutely it's actually
it's turning 10 years old right now.
So it's the more nights you create
that we have for look management
and look assessing in the market
and we're really happy to see that the sites project
that started 10 years ago.
It's adoption,
you can see continuous growing ecosystem
from a plugging perspective
and companies a lot in the technology that is really great.
It's very overwhelming and actually I'm really happy
to take this project and continue working with companies,
individuals and right now what is the position
where we are at now with Fluentd.
So if I put a roadmap,
it's like one of the pains that people is facing
not because of the tool because people has everything.
They has more data and more micro services.
That their system are scaling up.
It's like about performance, right?
And performance is creaking up if you're slowing
down the data processing actually you're not getting
the data at the right time where you need it right?
Nowadays people needs real bank queries, real time analysis.
So from a blended perspective we are going to focus a lot
on everything that is about performance.
I would say for this year and maybe the other one
I would say that we want to see many new futures
around Fluentd itself as a project
so it would be mostly about back fixing
and performance improvements.
>> Yeah I definitely want to dig in with you on the data
and logging challenges around Kubernetes
especially with end to end workflows and you know,
there's the different environments
that sits in the middle of but first before we get there
just take a minute to explain to the folks not that savvy
with Fluent Bit what is Fluent Bit real quick?
Explain what it is.
>> Okay, so I would start with a quick story about this
so when we started Fluentd we envisioned that
at some point I'm talking about six years ago right?
All this IT train or embedded or H will be available
and for that (indistinct) too heavy, right?
If you have a constraint barrier name
or you want to process data in a more faster way
without all the capabilities at that time
we say that Fluentd might not be suitable for that
so the thing is okay Fluentd was not longer
like a single software, right?
We wanted to say Fluentd is an ecosystem, right?
And as part of the ecosystem we have ACK
where people can connect applications with Fluentd
but also we say, we need like a Fluentd
but that could be lightweight and faster.
Fluentd is 3D, right?
And the critical parts can see
but since it's written Ruby of course
there's some a person calls on how do you process the data?
How much you can scale right?
So let's say if you're going to dig into embedded
or small constraint environment
let's write a similar solution but in C language
so we can optimize some memory,
we can optimize all these kind of needs will be fixed right?
We started the spread called Fluent Bit
and Fluent Bit is like nowadays
like a lightweight version of Fluentd.
It has started for the model Linux
but after a few years people from the Cloud Space
I'm talking about Containers, Kubernetes they started to ask
for more features for Fluentd because they had Fluentd
but also they wanted to have Fluent Bit on it
because of it was likely.
And nowadays we can see that Fluentd
is established in the market
and Fluent Bit we're getting around
2 million downloads every single day.
So now when they the attraction of the breaks is incredible.
And it's most of the used
to collect logs from the files, from system B
and from Mosul coordinators environment, disabled process
all this information in meta data
and solve all the problem of how do I collect my data,
How do I make sure that the data has the right context,
with the data and then able to deliver this data,
So that central place like a flood provider
or any kind of storage.
>> That's great.
And I love, I love the fact that it's written in C
which kind of gives the, it, I'll say it more performance
on the code, less overhead, get deeper, closer
and people know, know C it's it's high performance,
a quick, quick stats.
So how old is the project?
Fluent Bit?
What version are you on?
>> Eh, Fluent Bit, it's I'm, I'm not sure
if it's turning six or seven this year.
but, it's likely six
>> John: So it's been around for a while.
>> Likely six.
Yeah, yeah, eh, we just released this this week
1.7.3, right?
We had gone more than a hundred releases
actually the release cycle leaps pretty fast.
Sometimes it had releases every two, three weeks.
So the duration, they the Cloud Nation system is quite fast.
People want and more feature, more fixes
and they don't want to wait for a couple of months,
for the next release.
They want to have the container ematch right away
to test it out and actually stamps that we, as a project
we work with most providers like AWS, eh Microsoft, Azure,
Google Cloud platform, the demand for these fixes,
and improvements are in a weekly base.
>> You guys got a lot of props.
I was checking around on the internet.
You guys are getting strong, um, reviews on logging
for Kubernetes, with the couple of releases ago
you had higher performance improvements for you know,
Google, AWS, log DNA, Postgres equal
and mother, other environments.
Um, but the question that I'm getting,
and I'm hearing from folks is, you know,
I have end-to-end workflows and they've been steady.
They've been strong, but as more data comes in
and more services are connecting to it
from network protocols to other cloud services,
the, the complexity of what was
once a straight straightforward workflow end to end
is impacted by this new data.
How, do you guys address that?
How would you speak to that use case?
>> Well, for, for us, data, we have taken that approach
and leader for us is agnostic on the way
that it comes from, what did it comes from?
And then the format that comes from, for example
if you talk about Bitcoin, use the stasis that we have now
the second data from different formats
every single developer use the all different format.
Come from different channels, distinctly by the system
or another services.
So it is very, it's very different, how we get this data.
And that is a big challenge, right?
How do you take data from different sources,
different format, and you try to unify this internal
and then if you're going to talk, for example
to Elasticsearch, Elasticsearch, uses
(feedback)
you're going to talk to CAFCA,
they have their own binary protocol.
So we are kind of the backbone, that takes all
the data transfers the data and try to adapt, eh,
to the destination expected data,
but from a thing perspective.
Yeah, it's really challenging.
It's really challenging also that nowadays, so
two years ago, people was fighting processing, I dunno
5,000 messages per second, but then one day
They want 10, 20, 40,000.
So if I'm going to conduct a perspective, yeah
there are many challenges.
And I think, and I think that the teamwork
from the maintainers and with companies
has, has provided a little value, a lot of value.
And I think that the biggest proof here is
is that the adoption
>> John: Yeah.
>> Right, adoption, and, you get adoption, you have more
likes reported, morning chancellor requests.
>> Alright, so if I get this right.
You got different sources of data collection issues.
If you will, on the front end.
And then you've got some secret sauce
with a Bit Fluent, I mean, inside the Kubernetes clusters,
um, and then you deliver it to multiple services
and databases and cloud services.
Is that, is that right?
Is that the key, the key value?
Is that the, is that the key value proposition?
Did I get that right with Fluent Bit?
>> Yeah. I would say most of the technical implementation
that they're, off the value of the technical implementation.
I would say that it's two words being the vendor neutral.
Right?
So when you come to, go to the market
and you go to the, talking to bank institution, hospitals,
or any big company, right?
Most of them are facing this concept
of vendor lock-in, right?
>> John: Yeah.
They use a vendor database, but you have to get married.
So they're tooling, right?
And I'm not going to mention any vendor name, right?
>> You can name names, go ahead.
>> But actually it's pretty flat, well, for, for example.
The business model of this company that started with S
and ends with Splunk, for example is,
did you pay us much money?
So you pay as much money compared to the data
that you just, but the default tools ingests the full data.
But in reality, if you go to the enterprise, they say, yeah.
I mean, just in all my data plan or X provider, right?
But from the hundred percent that I'm ingesting
which I'm paying for, I'm just using the service to query
at least 20% of that theater,
so why I'm ingesting this 80% extra?
I, it, right.
That's why I want to say, this is real use case
based language is really good for queries analyzing data.
But they said, yeah, 80% of my data is just our tech data.
And we need it, maybe in a couple of months
so just I went to send it to Amazon.
Or any kind of other, eh, type service.
So users, the value that says is that I want to
have a vendor neutral pipeline, which me as a user, I went
to this site where, where to send the data when to send it.
And also I can control my bills, right?
And then since that is the biggest value.
So you can go to the market, they would find, eh,
maybe, all the tools for logging or tools for metrics
because there's a ton of them.
But I, I think that not all of them can say,
we are vendor neutral, not all of them can offer
this flexibility to the use, right?
So from a technical analyst performance
but from an end user, it's been the neutrality.
>> Okay, So I have to ask you then here
in the CNCF projects that are going on
and the community around um, um, Fluid Bit.
You have to have those kinds of enhancements integrations,
for instance for not only performance improvement,
but extensibility.
So enterprises they're, they want everything, right?
They make things very complicated,
>> Eduardo: Yeah.
>> They have very complicated infrastructures.
So if they want some policy they want to have
data ingestion policies or take advantage
of no vendor lock-in.
How has the community responding?
How, did, what's your vision for helping companies?
Now you've got your new venture,
and you've got the open source project.
How does this evolve?
How do you see this evolving Eduardo?
Because you know, there is a need
for use cases that don't need all the data,
but you need all the data to get some of the data, right?
So you have a new paradigm.
>> Yeah.
>> Of coding and you want it to be dynamic and relevant.
How do you see this evolving?
>> Yeah,
actually, I'm going to give you some spoilers, right?
So some news before we partake.
Yeah, so end users have this a lot of problems,
how to collect the data, process the data and send the data.
We just solved that, right?
Performance is a continuous improvement, right?
Because you have always more data, more formats.
That's fine, but one critical thing that
people say is eh, is eh, eh I want to put my business logic
in the pipeline, so think about this.
If you have to embed, where the platform for data, right?
But we also provide capabilities to do in data processing
because a you can grab the data or you can
do custom modifications over the data.
When one thing that we did like a, a year,
two years ago is we added this kind of stream
processing capabilities.
Kind of a sequel for CAFCA, but we have our own
sequel engine in Fluent Bit.
So when the data is flowing, without having any database,
any index or anything, we can do data aggregation
you can take, you can put some business logic
on it and assess for all the data that matches this pattern,
send it to a different destination.
Otherwise send it to CAFCA, Splang, or Elastic
So we have, this is what we have now
is assessing capabilities.
Now, what is the spoiler and what we're going next right now
there are two major areas.
One of them is distributed, extreme processing, right?
The capabilities put this intelligence on the edge
on the edge, I'm referring to, for example
um a Kubernetes know, right?
Well, I've got a strained environment right
a Kubernetes on the edge is something that is is going on.
There are many companies using that approach,
but they want to put some intelligence,
and data processing where the data is being generated
because there's one problem.
One is we'll have more data
and you want to put it that data.
You have to wait and to centralize all the data
in the database of your service.
And there's a latency, right?
Meaning sometimes hours because the data
needs to be indexed, but what about if you have
a hundred of notes but each one is a really random fluid
why you don't run the queries there.
That is the one of the future, just a, that we have.
Um, well, now talking from the challenges,
from the employer perspective is people says,
okay I love this pipeline.
I noticed flowing better, luggable architecture
but the language, C, eh it's not my thing, right?
I don't want to code in C, nobody likes C.
Now we are honest about that, and we are many buzzwords
about security and lots of stuff, which is true, right?
It's really easy to mess up things in C, right?
Eh, so, and we said, okay, so now our next level is like
we're going to provide this year, the ability to
write your own plugins in Westland web assembly.
So with a web assembly interface,
you can run with your own plugins
in Go Rust, or any kind of WebAssembly support language
and translate that implementation
to native Watson's, that Fluent Bit will understand.
So C as a language won't be with long, any longer
upload it for you as a developer or
as a company that wants to put more business logic
into the pack.
That is one of the things that are coming up.
I really, we already have some POCC,
but they're not ready to show.
So maybe we can stick something for
Kube Con U.S. at the end of this year.
>> Well, great stuff, by the way, from a C standpoint, us old
old timers, like me used to program in C and not
not a lot of C courses being taught.
But if you do know, C, it's very valuable, but again
to your point, the developers are, are
are focused on coding the apps, not so much the underlying.
So I think that's, that's key.
I will like to ask you one final question
Eduardo, before we wrap up, how do you deploy a Fluid Bid?
What's the, is it, is it you putting it inside the cluster?
Is there a, is it scripts?
What's the, what's the architecture real quick.
Give us a quick overview of the architecture.
>> Fluid Bit it's not just for a classroom.
You can run it on any machine windows, Linux, VM.
Yeah, it doesn't need to be a Kubernetes class, right?
When we created Fluid Bit, Kubernetes was quite in line
at the same time, eh, so if you talk about Kubernetes
you deploys as a game on set, and a game on set
to sprig a match up of pog
that runs on every note like an agent, right?
Or, eh, Japan running as a service on any kind of machine.
Oh and one thing before we wrap up,
I just missed to mention something that
from the spoilage part
because I get to, we're having many uses,
is, eh, Fluentd used to be mostly for logging, right?
And in Fluent Bit, its specifically project.
We've got many people from years ago saying, you know what
I'm using my agent for logging to embed,
like, having my agents for metrics?
And sometimes this is quite heavy to have multiple agents
on your agent.
So now Fluent Bit is extending in capabilities to deal
with native metrics, right?
Um, the first version will be available.
Well, this week on KubeCon, right?
Where were we able to process post matrix for application
metrics and send them to Prometheus with open metrics
format in a native way.
So we extended the clinical system to be a better citizen
with open metrics.
And in the future, also we pay open telemetry
which is a, a hot thing that is coming out on this month.
>> Everyone loves metrics.
That's super important, having the data is really
really important as day two operations and get ops.
All this stuff is happening.
Eduardo, thank you for coming on and sharing the update
and congratulations on the new venture.
We'll keep following you and lookin' forward
for the big launch, but Fluent Bit looking good.
Congratulations, thanks for coming on.
>> Thank you so much.
I hope you enjoyed your conference.
>> Okay it's theCUBE's coverage
of Kube Con '21 Cloud Native Con '21 virtual.
Soon, we'll be back in real life at the events
extracting the signal from the noise.
Thanks for watching.
(upbeat music)