In this video, you’ll see how to send VPC flow log data to Splunk using
Amazon Kinesis Data Firehose.
With this solution, you can create a Kinesis Data Firehose delivery stream with Splunk
as the destination, deploy a Lambda function for data transformation, and create Amazon
Virtual Private Cloud (Amazon VPC) flow logs to send data into Splunk.
Let’s review the architecture for sending AWS VPC flow log data into Splunk.
We have a Kinesis Data Firehose delivery
stream with Splunk as a destination via the HTTP Event Collector (HEC).
The Splunk endpoint could be on AWS infrastructure
or hosted in an on-premises data center.
An AWS Lambda function transforms VPC flow logs from their native JSON
to the required Splunk format.
Once set up, traffic in the identified VPC generates flow log data that is
sent to Kinesis Data Firehose and ingested into the Splunk destination.
Please review the pre-requisites shown here.
For the purposes of this video, we’ve already taken care of these prerequisites.
We’ll begin in Splunk, where we can see the Splunk Add-On for AWS in the left navigation.
Let’s go to Settings to create a new HTTP Event Collector token.
For this we’ll go to Data inputs.
We’ll give our HEC a name and description.
We’ll also enable indexer acknowledgment.
This is required for Kinesis Data Firehose destinations.
Next, we’ll select our input settings, beginning with the source type.
We’ll select aws:cloudwatchlogs:vpcflow.
This source type was made available by installing the Splunk Add-on for AWS.
Next, we’ll select the index where we want our Splunk data sent to.
Now we can review our settings and submit our HEC token.
Let’s note the token value, as we’ll need it when we configure our Kinesis
Data Firehose destination settings.
Next, let’s go to the AWS Management console and open the AWS Lambda page.
We’ll create a new function.
We can browse the serverless app repository to find the Lambda application we need.
We’ll use this processor from Splunk to transform data when streaming AWS
VPC flow logs to Splunk via Kinesis Data Firehose.
Let’s deploy the processor.
Let’s look at what we deployed.
The resources associated with the processor include the IAM role for the
Lambda function as well as the Lambda function itself.
We'll make sure to note the Physical ID of the Lambda function, as we will need to
provide it on the next step, when we create the Kinesis Data Firehose delivery stream.
Now, let’s navigate to Amazon Kinesis services.
We’ll create a Kinesis Data Firehose delivery stream.
First, we’ll specify Direct PUT as the source and Splunk as the destination.
Let’s provide a different name for the delivery stream.
Transforming records is optional.
Let’s enable data transformation and choose the Lambda function we created.
We’ll expand our buffer size to 1 MB
but keep the buffer interval to the minimum setting of 60 seconds.
Under Destination settings, we’ll specify our Splunk cluster endpoint.
For the authentication token, we’ll specify the HEC token value we noted earlier.
Under Backup settings, we’ll choose an existing Amazon S3 bucket
where failed events can be saved.
Let’s create the delivery stream.
Next, we’ll navigate to the VPC Management console to create our VPC flow logs.
We’ll select VPCs to see the VPCs that we already have in our environment.
We’ll select this one.
Let’s create a new flow log.
Let’s give our flow log a name and leave the Filter settings to the default
in order to capture all traffic.
We’ll reduce our Maximum aggregation interval to 1 minute.
Let’s change the destination where flow log data will be published to Kinesis
Firehose in the same account and then specify the name of our delivery stream.
We’ll keep everything else as the default and create the flow log.
Let’s view the flow log.
When the flow log is available, we can see it’s ID.
Now let’s go to Splunk and open Search & Reporting, where we can
verify that our integration is working.
We’ll search for the index that we specified in our HEC input settings.
We’ll also choose to show data from the last 15 minutes only.
Data is already coming into Splunk.
We can select the sourcetype field to confirm that VPC flow logs are the source of our data.
The integration has been successful.
You’ve seen how to send VPC flow log data to Splunk using Amazon Kinesis Data Firehose.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.