In this video, I'll introduce you to querying Loki logs in Oracle Private Cloud Appliance.
Loki is a log aggregation system which serves as the backend data source for logs.
Loki has a powerful query language that was inspired by Prometheus and extended for logs.
It's described on Loki's web site as being like "Prometheus, but for logs".
This basic design overview shows that we have various services like api-server, admin, network,
storage, and compute.
Loki does not use a pull model, so unlike Prometheuse, it does not scrap logs, Loki
pushes them.
Here we expound on one of these microservices, compute.
The compute container will write logs to this particular location.
We also have the log rotate here.
And we have the Fluentd side-car which is responsible for monitoring these particular
logs and would push these logs to Loki.
We also have the system-level logs like OS logs, layer0 logs, K8 logs, hypervisor logs,
and network logs which are gathered by the Fluentd daemonset.
Once data is Loki, use Grafana to get it back out.
So Grafana is the front-end visualization dashboard for looking at these logs from Loki.
I'm logged in to the Grafana dashboard for my Oracle Private Cloud Appliance.
My system name is blurred out for security purposes.
Grafana provides an "Explore" based workflow for querying logs.
But you can also build dashboards that contain your log content.
You can create metrics from your logs and actually have graphs of your log volume and
aspects of your logs.
There is also a logcli command-line interface to query Loki but this video focuses on the
"Explore" feature of Grafana.
I'll click the Explore button and the first thing you see is a Loki Cheat Sheet.
This is assuming your data source is Loki as shown.
Your other data source is Prometheus which provides a metrics selector and a PromQL cheat
sheet.
So ensure you have selected Loki as the data source before continuing.
The cheat sheet provides a few examples of how to query your logs.
As it says, you start by selecting a log stream from the log labels selector.
Labels in Loki are key value pairs.
The combination of every label key and value defines a stream.
So a stream is defined as the unique key-value pairs associated with a log line.
Before I start demonstrating Loki queries, I'll scroll down and show the rest of this
cheat sheet.
Notice that the log stream selector is written by wrapping the key-value pairs in a pair
of curly braces.
You can combine key-value pairs, seperated with a comma.
This query returns all log lines from streams that have both labels.
Once you have provided the stream selector, you can filter the log lines that are returned
in a number of ways.
I'll demonstrate filters later in this video.
Scrolling on down the cheat sheet shows examples of metric queries.
Whereas log queries return the contents of log lines, metric queries extend log queries
to calculate values based on query results.
As of this recording, the documentation for using the Loki query language is located at
https://grafana.com/docs/loki/latest/logql/.
There is a section in the documentation describing log queries and metric queries.
You can also search the Internet for 'Loki Cheat Sheet' which will return a number of
hits.
I'll now scroll back up and demonstrate entering a few queries.
I'll click the down arrow for the Log lables to view the available labels.
Each of these with exception of "_name_" label provides a list of possible values.
Here are the values for the fluentd_thread.
Here are the values for host, which includes 3 compute nodes and 3 management nodes in
this appliance.
Here are the values for job, which include admin, api-server, asr-client, and others.
This job label is good to use in queries because every service has a job.
I'll run my first query by scrolling down and selecting compute and the query results
are shown immediately.
By default, the time range is the last one hour but you can change this.
Another default is to only list the latest 1000 messages but you can also change this
by entering a different number and then click the Run Query button.
You can zoom in on a particular range by selecting the range on the graph.
The data is displayed along with the time range you selected.
Now I'll show an example of how to filter query output.
Say that I only want to see the compute logs for a specific fault domain, for example fault
domain 3.
Currently I'm showing 66 log lines.
I'll use a pipe equal sign and then provide the fault domain within double-quotes.
I'll click run query and now I only have 6 log lines.
Here are the operators that you can use for label matching and filters.
The search expression can be just text or a regular expression.
So in this query, my label is job exactly equal to compute and I'm filtering for log
lines that contain the string "FAULT-DOMAIN-3".
I'll scroll down a bit and show that you can click anywhere in a log line to provide an
expanded view.
This view shows the log labels key-value pairs.
The labels for this log line are host, job, namespace, and podname.
This also shows the parsed fields which include loglevel name, line number, message, and other
information.
The rid is the request id which is used for tracking the request through multiple services.
The last thing I'll demonstrate is using the health checker dashboard in combination with
Loki log queries.
I'm back to the main Grafana page and I'll open up the Platform Health Check dashboard
which is in the PCA 3.0 Service Advisor folder.
This dashboard shows a NOT_HEALTHY status for the network-checker at this time.
So I'll copy this timestamp then click the Explore tab to enter my Loki query.
For log label, I'll select job=network-checker and then apply the timestamp I copied from
the dashboard as a filter.
I'll click Run Query and I now have 13 log lines I can view to help determine the cause
of the not_healthy status.
This concludes this demonstration of querying Loki logs in Oracle Private Cloud Appliance.
Thank you for watching.
In addition to the Loki query language resources provided earlier in this video, please check
our additional information on Oracle Private Cloud Appliance in the Oracle Private Cloud
Appliance Documentation or in the Oracle Private Cloud Appliance Learning Library.