One of the standard methods for transferring log files
from one device to a centralized database is called Syslog.
This is a standard that you'll find
across many different devices, and if you're
installing a firewall or switch or a server,
you'll notice that there will be Syslog options
within those devices so that you could send that data to a SIEM.
This is a Security Information and Event Manager
and it's usually a centralized log server
that consolidates logs from all of these different devices.
When we send information via Syslog,
we're labeling each log entry into this Syslog destination.
There will be a facility code, which
is the program effectively that created the log,
and there will be a severity level associated with that log
entry.
If you look in a Linux device or device that
doesn't automatically have Syslog functionality, you
may see different daemons available,
such as rsyslog, that is the rocket-fast system for log
processing, you could see syslog-ng
which is a popular syslog daemon for Linux devices,
or NXLog, which is able to collect log
information from many different devices
and consolidate it on a single machine.
This is the front-end of a security information and event
manager, that has received these log files via syslog,
pass those files, and now we can view this information,
we can search through this data, and create reports
on everything stored in our database.
If you're managing a Linux operating system,
there are many different logs available on that device.
Some of them are specific to the operating system itself,
some of the logs are created by the daemons that
are running on that system or the applications
that you're using.
There is a standard format for storing system logs on Linux
in a special binary format.
This optimize the storage area and allows
you to query the information very quickly.
But you're not able to see it with a text editor
because it's in the special binary format.
Fortunately, Linux has a utility called journalctl,
which allows you to query the information that's
in that system journal and provide output on what
may be contained in there.
And you can search and filter on those details,
or view it as plain text.
This is a view of the output from the journalctl
and you can get an idea of information
about connections to SSD, here's in rsyslogd entry, where
syslog information has been received,
and you can look at other details
about authentications that have either failed or not
failed on the system.
One of the first statistics we often
want to gather from these log files
are information on the bandwidth that we happen to be using.
This is a fundamental network statistic
and one that is almost universal no matter what device you're
connecting to.
This shows you the percentage of the network that
has been used over time.
And there are many different ways to gather this metric.
You might use SNMP, the Simple Network Management Protocol,
or you could use other more advanced capabilities
such as NetFlow, sFlow, or IPFIX.
You could also use protocol analyzers or software
agents that might be running on a particular device.
Bandwidth monitoring is always a good first step.
It's good to qualify that you have
the bandwidth available to transfer information
for that application, because if the bandwidth has been exceeded
and you're running out of available space on the network,
then none of your applications are going to perform very well.
Another great source for data that
is in some ways hidden from us usually is metadata.
Metadata is data that describes other types of data,
and usually, metadata is contained within the files
that we're using on our devices.
For example, if you send or receive an email,
there is metadata within that email message
that normally you don't see.
There's information in the headers of that email,
that header information may show you,
which servers were used to transfer that email from point
A to point B, and you might want to be
able to see destination information as part
of that header in the email as well.
If you're using your mobile phone,
there's an extensive amount of metadata that could be stored.
For example, if you take a picture or store
video on your mobile device, it could keep in that metadata
the type of phone that was used to take that picture or the GPS
location where the picture was made.
If you're using a web browser to connect to a web server,
then there's metadata that's transferred back and forth
there as well.
For example, you could be sending your operating system
information, the type of browser that you're using,
and the IP address that you're sending it from.
And if you look into documents or files
that you store, for example, in Microsoft Office,
you may find metadata inside that document that
shows your name, your address, your phone number, your title,
and other identifying details.
Here are the headers that you normally
don't see when you're looking at your email messages,
and the metadata that's hidden inside.
You can see what IP address a message was received from,
and who received that message, you can see the return path,
another IP address where that message was received,
and other details that help you understand what path this
to go through the network, what validations were used
to be able to confirm this message
was really sent by that originator, and other details
about this email message.
NetFlow is one of these standardized methods
of gathering network statistics from switches, routers,
and other devices on your network.
This NetFlow information is usually
consolidated onto a central NetFlow server,
and we're able to view information
across all of these devices on a single management console.
NetFlow itself is a very well-established standard,
so that makes it very easy to collect information
from devices that are made from many different manufacturers,
but bring all of that information back
to one central NetFlow server.
This is an architecture that separates the probe
from the collector.
So we have devices on our network
that may be individual NetFlow probes,
or the NetFlow capability may be built into the network
devices that were using.
These probes are either sitting in line with our network
traffic, or they're receiving a copy of the network traffic,
and all of those details are exported to a central NetFlow
collector where you can then create different reports.
There are usually extensive reporting options
available on the collector, and we
can gather very long-term information
to be able to see trends and other details
about how our network is performing.
Here's a NetFlow collector front end
that shows the top 10 conversations and top 10
endpoints on our network, and it shows it
as it's relating to bandwidth.
We can also get a breakdown of all individual hosts
here as well.
And here's another summary of details
that shows the top five applications running
on our network, and what the top NetFlow sources might be.
A similar data flow standard is IPFIX.
This is the IP flow information export
which you can think of as a newer version of NetFlow.
It was one that was created and based on NetFlow version nine.
This allows us with some flexibility over what data
we would collect and what information would be reported
to a centralized server.
This is very similar to NetFlow, except we can customize exactly
what kind of data we'd like to receive from those collectors.
One of the challenges with collecting this network traffic
and creating metrics based on the conversations occurring
on our network is that it can take a lot of resources,
especially, if you're running a very high-speed network.
To be able to balance the available resources
with the need to view more statistics on the network,
this is sFlow or sampled flow, where
we're looking at a portion of the network traffic
to gather metrics on.
Because of the lower resources required for sFlow,
we can embed this capability in a number of our infrastructure
devices.
So the switches and routers that you're already
using on your network, may already
support sFlow functionality.
And although we're only looking at a portion
of the traffic going through, we can
infer some relatively accurate statistics
from these sFlow devices.
You may be able, for example, to view
video streaming and high-traffic applications,
by simply sampling a portion of that traffic
as the flow is active.
Here's an example of some of the statistics
you can gather using sFlow.
You can see the top 10 interfaces
by a percentage of utilization, we
have top 10 interfaces by total amount of traffic,
top 10 wireless clients by traffic, top 10 wireless
access points by client count, and other statistics as well.
And if you need to get detailed information of exactly
what's going over your network, then you
can use a protocol analyzer.
Protocol analyzers are commonly used
to troubleshoot complex application problems,
because they gather every bit and bite from the network,
and provide a breakdown of exactly
what's going across those particular network links.
You can also use this on wireless networks or wide area
networks as well.
You're able to see information such as unknown traffic, that
may be going across the network, you
can provide packet filtering so that you
can view exactly the information you're looking for,
and of course, the protocol decodes on the analyzer
will give you a plain English breakdown
of exactly what traffic is traversing the network.
Here's a screenshot from an analyzer.
On the top, we can see a packet by packet breakdown
of delta times source IP addresses, source port
numbers, destination IP addresses, destination port
numbers, protocols, the length of data in the packet,
information about the packet, and then
underneath all of that, we have a detailed breakdown
of every single packet that's going through the network.