Today we are going to talk about Logstash,
and what we are doing to make it easier to adopt the Elastic Common Schema.
We’ll provide primers for both Logstash and ECS,
show how ECS can be used from Logstash plugins,
and provide steps for opting OUT of the transition.
Hello, I am Ry Biesemeyer
one of the Principal Engineers working on the Logstash project.
As you likely know, Logstash is a tool that facilitates
user-defined pipelines for transforming and enriching a stream of events.
It is a part of the Elastic Stack,
but events from those pipelines are not CONSTRAINED to the Elastic Stack.
Our users define their pipelines using a domain-specific language to connect
input plugins with codec plugins, through a sequence of filter plugins to output plugins.
It's really all about the plugins.
But what are plugins?
Logstash input plugins create events, often with the help of Codec plugins
Codec plugins either transform an arbitrary sequence of bytes into a structured event
or the other way around
Filter plugins transform and enrich batches of events
And Output plugins persist batches of events elsewhere
Logstash allows users to tie any number of these plugins together in a pipeline,
in a manner that addresses their specific data needs.
Here we have a simple pipeline definition.
It includes a single TCP input plugin
...that uses the CEF codec to create events from inbound data...
...a couple filters that batches of events are conditionally routed to in sequence...
...and a single Elasticsearch output plugin.
In the real world, pipeline configurations can grow to be thousands of lines
and can invoke a sequence of hundreds of plugins with layered conditional logic
to mutate and enrich events before eventually routing them to outputs
like Elasticsearch, S3, or even other Logstash pipelines.
And while Logstash is a part of the Elastic stack,
we know that not every Logstash pipeline in the real world routes to Elasticsearch.
We're here for those too.
Logstash is powered by its plugin ecosystem.
Plugins are versioned and packaged independently,
and are completely independent of the Elastic Stack release versioning.
This provides a number of advantages:
We can release fixes to plugins more rapidly than the Elastic Stack release cadence
Users can also build and maintain their own plugins
that use the same APIs as the plugins we bundle with Logstash core.
Historically, there was no single standard for how a plugin was supposed to enrich data.
Many plugins originated as donations to the Logstash project from outside sources,
and the more widely they were adopted, the harder they became to change.
While some plugins require explicit configuration for absolutely everything,
many implicitly populate fields on the event in their own way.
This can lead to schema conflicts when certain plugins are used in combination with each other,
or when they target an Elasticsearch index that also contains data from another source.
As the Elastic Stack has evolved, we have discovered that when data from many sources
fit into a common schema, each source becomes a value multiplier.
For example, we can correlate events indicating high latency to a destination address
with other events that have CPU or memory metrics about that host.
We leverage shared schema heavily in our Observability dashboards.
The Elastic Common Schema is a community-driven schema
that allows us to encode the semantic MEANING of a field into that field's name,
so that when two or more data sources describe the same THING they can be used together.
We have long wanted to move into a world where
a brand new Logstash user gets ECS-compatible events
without having to manually remap each and every field produced by their incoming data.
But more importantly, we needed to avoid breaking existing pipelines.
This is part of why Logstash is a little late to the game of adopting ECS.
I would like to show you three simple demos to set a baseline for behavior.
We're looking mainly to show complexity, and these scripts will be made available.
Here we will be parsing log lines encoded with in the Common Event Format (CEF)
CEF encodes a fixed number of fields at the beginning of each line,
and provides a framework for generic key/value pairs with shorthand naming.
Our example input is very long, with many extension fields.
Our pipeline config is pretty straight-forward: Our regular input uses the CEF codec,
and the rest is boiler-plate to make it work with the demo capture tool.
And we run Logstash...
As you can see, the resulting structured event is flat and really CEF-centric.
It won't mix well with other data sources, and has ECS conflicts like the top-level `host` field.
In ECS, the top-level `host` is an object with many attributes
-- ip, name, geographic data, and more --
the attribute populated here really should be `[host][hostname]`.
For the next demo, we have a simple GeoIP enrichment.
I've set us up with the simplest of inputs: a top-level host that has an IP address
And our pipeline is pretty straight-forward too:
we have a single geoip filter instance, configured to enrich the host’s IP.
This results in an event with a top-level `geoip` field, which is a little problematic.
It does not contain any context as to what it is referring to --
and while it includes a GeoPOINT structure for the coordinates,
the shape of the remaining fields isn't _quite_ ECS geo-type either.
Finally, we have a simple Grok extraction.
We have two inputs this time: an HAProxy log and a generic access log.
This represents a pipeline that is handling inputs from multiple applications and systems,
which is common in the real world.
We can scroll and see that the HAProxy log line is pretty long
Our pipeline includes a Grok filter that matches for either
The result is an explosion of top-level fields including a lot of useless ones
like individual hour, minute, and second fields from the HAProxy timestamp.
We can scroll down to the other event and see that it too is flat
...and has ECS conflicts like that troublesome top-level `host` field
In Logstash 7, it is certainly has been possible to transform a pipeline of events
so that each is ECS-compatible, but it hasn't been pretty.
Because many plugins IMPLICITLY violate ECS
by populating fields without explicit instructions,
ensuring your pipeline produces ECS-compatible events
has traditionally entailed a LOT of cleanup.
Let’s take a look.
We’ll repeat the CEF demo, but with manual ECS remapping this time.
We’re using the same CEF-encoded log line from last time
But this time our pipeline config is a bit more complex.
We add filters to clean up after the STDIN input
More filters to remap the CEF header fields into a ECS-compatible fields
More Mutate filters to remap the various extension fields from our input
And we also add some Date filters
to make sense of the timestamps in the context of the event’s timezone.
As you can see, the pipeline has gotten pretty complicated.
But when we run it through Logstash...
...we end up with a nicely nested ECS-compatible event...
...complete with CEF-specific namespace for CEF-specific fields.
Next up, we’ll repeat the GeoIP enrichment in an ECS manner...
...taking the same simple input...
...and again we create a more complex pipeline to do the remapping.
We target the results of our GeoIP lookup to a metadata field
so we can avoid polluting the top-level namespace
And then we remap the fields...
...and even though @metadata fields aren’t included in our output,
...we clean up after ourselves.
The result is ECS “geo” object,
properly nested underneath the `host` whose `ip` we looked up.
And finally, let’s repeat the Grok demo, with manual ECS this time.
Again, we use the same inputs as last time.
First, we clean up metadata from our input
and move the “message” out of the way to `[event][original]`
because we know that one of our grok patterns captures to `message`.
But then things get complicated really fast.
The cleanup is pattern-specific, so we can no longer use a single Grok filter
...to “fall through” a sequence of patterns.
And when I say complicated, I mean it.
We have to re-map a LOT of fields...
...remove the useless captures...
...and do a bunch of conversions.
And that is just for the first pattern.
We then need to attempt other patterns if we haven’t succeeded yet.
And we have to clean those up too
And then if we succeeded at making one match, we need to parse the timestamp too.
And that is just for two patterns.
The wider the variety of logs the pipeline handles, the more complex this gets.
But when we run our input through this pipeline...
The effort pays off and we end up with nicely-structured events...
...including namespacing for vendor-specific metrics...
And all of the appropriate top-level ECS fields.
As you can see, even with these simple cases, complying with ECS is possible
but it requires you to create verbose pipeline definitions.
It makes those pipelines harder to maintain
and introduces more opportunity for human error.
We wanted to obviate the need for this cleanup.
We wanted ECS-compliant pipelines to be simpler and more readable,
to eliminate the overhead of ECS without forcing our users into a specific schema.
We know that ECS-by-default is the right solution for new users and new pipelines,
so it will be on-by-default in Logstash 8.
But we also knew that our solution needed to work for our existing users
and their body of pipeline definitions, providing
a simple opt-out for users who maintain complex pipelines...
...that they can't afford to change, AND
a process for migrating existing pipelines for those that desire to do so.
Importantly, we wanted to make this INDEPENDENT of Logstash upgrades
so that it didn't become a barrier to upgrades.
In Logstash 7.10, we added an option called `ecs_compatibility` to the plugin API,
powered by a new setting `pipeline.ecs_compatibility`
that can be specified for an individual pipeline in `pipelines.yml` or globally in `logstash.yml`.
This API bridges the gap between plugin maintainers
and our users who are defining pipelines.
Logstash users can USE this API to specify in which mode a plugin should be run
for an individual plugin instance, for an entire pipeline, OR globally; AND
plugin maintainers can query this API during plugin initialization
to determine the effective MODE in which the plugin should be run.
We then released a support adapter,
which provides a simplified implementation of this API
so that when a plugin is run on an older Logstash, the plugin can continue to work
and since that older version of logstash doesn't provide the pipeline-level setting,
it is simply off by default.
This approach eliminates the risk and overhead of maintaining
separate implementations in separate branches
in each of dozens of separate plugin repositories.
We also added tooling to the support adapter
to make side-by-side implementations in plugins easier to author, maintain, and test.
Then we got to work implementing ECS compatibility modes in a host of plugins.
There were a few different classes of problems to solve.
Some plugins needed to change the metadata shape
The UDP and TCP inputs record metadata about the host and port that sent the payload
The File input records metadata about the source file's path
Kafka & RabbitMQ add metadata about the topic or queue that we received the event from
and so on.
Some Plugins needed to change their implicit schema
The GeoIP Filter should use ECS's "geo" type
and should also target the result to somewhere meaningful
The CEF Codec should use the Elastic-standardized CEF mapping
(same as Beats and Agent)
Some Plugins needed to add an optional target,
to allow users to declaratively place the results somewhere specific
instead of populating top-level fields that are likely to clash with ECS.
The JSON and CSV filters and codecs populate fields defined in the payload
that aren't necessarily known when the plugin starts up.
The JDBC and Elasticsearch inputs take the results of a query,
and a `target` can be helpful here too.
The Grok filter was in a class of its own.
It needed a massive overhaul of all of its sub-pattern captures.
We had to understand the semantic MEANING of each capture
so that we could appropriately route it to the right ECS field.
This was no small undertaking, and
my teammate Karol Bucek deserves special recognition for his effort.
When solving these problems, we implemented ECS Compatibility
alongside each of the existing implementations.
Each plugin has automated tests that ensure
it is capable of behaving EXACTLY as it did
before we introduced this optional mode.
Our team identified 75 Elastic-supported plugins for inclusion in this effort.
We can use these plugins in ECS Compatibility mode right now in Logstash 7
to define simple ECS-compatible pipelines without the complex remapping.
The following examples require opting-in for the pipeline,
which can be done by setting
`pipeline.ecs_compatibility` in the `pipelines.yml`
We’ve set the setting, so let’s run through the same demos
and get ECS compatible events more simply.
We start again with the CEF example.
We have the same input log line
And our config is simple again.
Just the CEF codec in the input and the boiler-plate
to make it work with my demo.
When we run Logstash...
...we end up with an ECS-compatible event...
...logically grouping the observer, the agent, the event, and more.
We’ll do the GeoIP enrichment again,
this time with ECS Compatibility globally enabled...
The same simple input as last time...
...but our config is simple again...
...we run Logstash once more...
...And the enrichment is ECS-compatible:
...a geo-object nested inside the host whose IP we looked up.
And finally we’ll do some ECS-compatible Grok parsing...
We use the same inputs as last time...
And our pipeline config is back to being pretty simple.
We can regroup the separate grok matches back into a single filter instance,
set our `[event][original]` with the inbound message,
and don’t need to worry about much else....
When we run the pipeline...
...the results are ECS, just as we expect them to be...
...without all of the complexity of mapping it ourselves.
As you can see, we can have ECS, simply.
In Logstash 8, ECS will be on-by-default.
This means that each and every plugin that implements an ECS compatibility mode
will operate with ECS unless configured otherwise.
We want our users and customers to be confident in their upgrade path to Logstash 8 and beyond.
This is a pretty major change to how many of our plugins work.
For our customers and users, who are maintainers of Logstash pipelines, what does this mean?
When it is time to upgrade Logstash 8.0, we encourage you to
first upgrade through the final minor of the 7.x series: 7.17.
Because minor upgrades within the 7.x series are non-breaking,
this step will ensure that you are running the latest versions of all plugins,
and will give you a chance to resolve any deprecation warnings
before doing the major-version upgrade.
Beginning with 7.10, we emit an entry to the deprecation logger when a plugin attempts
to determine the ECS compatibility mode and the pipeline has not defined it.
We want you to KNOW when your existing configuration is in danger of breaking
at a future major upgrade, and to give you a chance to fix it first.
If you have a pipeline that is complex,
and that you can’t risk changing or afford to migrate,
you can opt-out easily with one simple configuration.
The `pipeline.ecs_compatibility` can be set to `disabled`
for an individual pipeline in the `pipelines.yml`
or globally for all pipelines in the `logstash.yml`
This LOCKS IN the existing behavior and resolves the deprecation warning.
If you would like to migrate a pipeline to use ECS
and gain the benefits of the common schema, it may be helpful to instead
resolve the deprecation warnings one plugin at a time in a development environment.
To do this, you will need to understand how running in an ECS Compatibility mode
changes the behaviour of each individual plugin
the documentation for each plugin has tables and callouts showing differences
You will also need to resolve any "downstream" effects in their pipeline.
For example:
if you configure an instance of the CEF codec to run in ECS Compatibility mode,
a downstream GeoIP filter that enriches the IP address of the `deviceAddress`
that is coming from CEF will instead need to enrich the `[host][ip]`.
Similarly, if you have a Kibana visualization that relies on the `deviceAddress`
it will also need to be updated to use the host’s IP instead.
Make one change,
resolve its downstream effects,
rinse and repeat until all plugins for the pipeline are running in ECS.
Once the deprecation logs have been fully resolved in this manner
and you are running the latest 7.17 available,
you can specify `pipeline.ecs_compatibility` for the entire pipeline
and remove the individual per-plugn `ecs_compatibility` declarations.
We know that using a common schema for data from disparate sources is
really the key to unlocking multiplicative value,
and hope that you find the Elastic Common Schema to be useful.
We are moving toward providing a better default experience for new users and new pipelines
with regard to the Elastic Common Schema,
and we are doing so without compromising the experience of those of you who own
and maintain complex pipelines or who don't target Elasticsearch at all.
We are providing the tools to make the migration to ECS independent of Logstash upgrades
so that our users have the freedom to either lock-in their existing behavior
or to begin reaping the benefits of ECS during the 7.x series.
The demo scripts are available at the URL on your screen and in the video description.
I would like to thank the Logstash team for their effort.
We had a giant surface area to cover, and the team worked together to make it happen.