Welcome to "This is My Architecture"
I am Kiriyama from Amazon Web Service
Today, we have Mr. Mizutani from Cookpad
Thank you for coming today
Thank you
First, tell us about Cookpad
Cookpad is a platform for sharing
cooking recipes and ideas
We have around 100 million users around the globe
So protecting the information of these users is
extremely important for us
My work involves protecting user information
and protecting this architecture itself
So your role is to protect the Cookpad service and the security of the users
Yes, exactly
Could you explain what you mean
by that this architecture is a tool to accomplish the purpose?
It was structured to monitor security logs
It was structured to help those who monitor security,
or those who respond to incidences
For instance, a member of the internal security operation center
uses this architecture
So this solution was made for the security members
How does it work?
This architecture consists of three parts
The first is where the log is collected
Next, the log is retained, then analyzed These are the three parts
Regarding the first log collection
It is equipped with various functions for log collection
From the EC2 instance, Fluentd collects syslog messages
and saves them in S3 via Firehose
For several managed services like Cloudtrail,
logs can be sent directly to S3
So logs are saved by using these functions
From third party services, logs are collected using Lambda
Then logs are saved in S3 via here
I can see that different types of logs are handled
What about the second part?
The second part is log retention
All logs are saved in S3 Using S3's life cycle policy,
we control the entire log retention period
All logs are stored in S3
and managed by the S3's life cycle policy
What about the last part?
The last part is the log analysis
The short term objective is
to find alerts in logs and notify them
using lambda function
For alerts that we find, we notify Slack
However, as a mid- to long-term objective, in order to be able to search the log
Lambda function sends the log to Graylog
Graylog is an open source log search engine
It uses Amazon Elasticsearch Service as a back-end.
The ultimate goal is to make it possible to do long-term search for these logs
It is equipped with functions for auditing
This is made possible by Amazon Athena
Thank you I was able to understood the big picture
Of these, which part is most innovative?
Security engineers need to respond to alerts as quickly t as possible
Based on SLA in a security service that is managed generally,
Regarding the time to process the log,
we want to keep it around one to two minutes
However, if a system that processes logs real-time
tries to build a system to handle a significant amount of logs,
it tends to get extremely costly
But this architecture has the characteristic of achieving both
the low cost and decreased latency period, or log processing time.
You mentioned low cost
Where does the cost merit lie in this architecture?
This architecture alone handles around 300 million logs per day
This is equivalent to 270GB
The cost of the Lambda function that handles these logs
is kept as low as about two dollars per day
Around two dollars per day is incredible
Compared to the general log monitoring solutions,
what is exceptional about this solution?
Regarding general log management tools,
in general, before collecting logs
a log schema needs to be defined
But for this architecture, in the log collection part,
we can collect without defining the log schema
This is because when S3 saves the log
it can save it without concerning schemas
By not having a schema, various logs are easier to handle
Finally, are there any plans for future expansion of functions?
Yes As we have collected security logs,
we want to create a system that detects abnormalities
using Amazon SageMaker
That is amazing Thank you for sharing
Thank you so much
Today we had Mr. Mizutani from Cookpad
discuss security monitoring solutions
Thank you for watching "This is My Architecture"