Hello, Welcome to Oracle Linux how-to video series.
This video is to help you with configuration of kdump on an Oracle Linux server.
All steps and procedures explained and performed in this video are based on Oracle Linux release
7.
Kdump is the Linux kernel crash dump mechanism that allows us to save the contents of the
system memory for analysis.
Oracle recommends that you enable the kdump feature as it helps Oracle Linux support to
find the root cause whenever a server panics or crashes.
In the event of a system crash kdump creates a memory image also called the vmcore that
can help in determining the cause of the crash.
Enabling kdump requires you to reserve a portion of system memory for exclusive by kdump
This memory is unavailable for other uses.
kdump uses kexec to boot in to a second kernel whenever the system crashes.
kexec is a fast boot mechanism which allows Linux kernel to boot from inside the context
of a kernel that is already running without passing through the bootloader stage
Pre-requisites for kdump are that the kexec-tools package be installed on the server
And a certain amount of memory be reserved for kdump
This is for kexec to boot in to the secondary kernel
And a path be specified for the memory dump to be stored and it should have sufficient
free space.
It is also necessary that the kdump service be running
kdump mainly uses 3 files The grub configuration file.
This is where we specify the crashkernel parameter to reserve memory for kdump
Next the /etc/kdump.conf file which is the kdump configuration file
This is where we set the location for storing the dump file and also the filtering level
for corecollector which will be the makedumpfile command and the default behavior to take if
the dump fails And it also relies on the kdump initramfs
file which will be present under the /boot directory
Let's see a quick demo on how to configure kdump and generate a dump from it
Firstly we need to make sure that the kexec-tools package is installed on the server
We see that it is not installed So let's go ahead and install it
Now the package is installed Next we have to verify the crashkernel parameter
in the grub configuration file This is done via the /etc/default/grub file
since we are using grub2 Here we set the crashkernel parameter and
reserve a certain amount of memory for kdump.
The value for crashkernel can be either auto or a specific amount of memory
We can see that it is set to 256M at an offset of 64M on this server.
For larger servers with higher amount of physical memory, for example 128GB of physical memory,
it is recommended to set crashkernel to 512M and similarly for even larger servers a higher
amount of memory needs to be reserved for kdump.
Once the crashkernel parameter is added to the /etc/default/grub file the grub configuration
file needs to be rebuilt.
For that we use the grub2-mkconfig command We see that the grub entries for each installed
kernel was updated.
Let's see the grub configuration file.
We see that the crashkernel parameter was added to the kernel entries in the grub configuration
file.
Once the grub configuration is updated, the server needs to be rebooted for the changes
to take effect.
I have already rebooted the server previously.
Next we have to configure kdump service which is done via the /etc/kdump.conf file.
Here we see that the path parameter is set to /var/crash
This is the default path.
This is used to specify the location for the memory dump to be stored.
It is essential that the location have sufficient space.
Next the corecollector needs to be configured and will be using makedumpfile as our corecollector
For more details on the makedumpfile please refer to the manual page.
The -d option is for the dump_level parameter.
Dump level specifies the unnecessary pages for analysis.
These pages are not copied to the memory dump The message_level parameter is for setting
the message types that need to be collected Users can restrict outputs printed by setting
the message level.
Default value for message level is 7.
Here we are setting it to 1.
After making changes to the kdump configuration file the kdump service needs to be restarted.
While restarting the kdump the initramfs file is rebuilt to
update the changes.
It will be present under the /boot directory.
Let's see the file.
Here we see that the kdump initramfs file is generated
To ensure that the kdump configuration is proper and a vmcore will be generated when
the server crashes, we can trigger a manual crash of the server and verify that the dump
is being generated.
We do this by echoing a character 'c' to the /proc/sysrq-trigger file.
Sysrq is a magic key and these keys are combination of keys which can be pressed and the kernel
will respond to it no matter whatever else it is doing unless it is completely locked
up.
Echoing a character 'c' to the sysrq-trigger file causes the server to crash by a null
pointer dereference.
Before triggering a crash make sure that a downtime is available for the activity in
case the server is a live production system.
Let's crash the server.
Now we are in the serial console of the server.
We can see that the dump is being collected and that is has completed collecting it.
Once the dump is collected the server goes for a normal reboot.
The time required to collect the dump varies based on the amount of memory that needs to
be dumped.
Now we see that it is rebooting and it is up now.
Let's check the path where we had configured the dump to be collected.
Let's check under /var/crash We see that a directory with timestamp was
created.
There are two files under it, that is the vmcore file and the vmcore-dmesg file.
We see that it is a data file.
And the size is considerably small as this is a server with only 4GB of memory.
So we see that the kdump configuration was successful and a memory dump is generated.
In case the server crashed at any time, we can be sure that the vmcore will be generated.
You can upload the vmcore file to Oracle support for analysis.
Here are some useful links and few my oracle support knowledge documents.
The knowledge can be accessed by logging in to support.oracle.com
Please go through them for more information on kdump and known issues.
That ends our short video on configuring kdump on an Oracle Linux server.
Thank you for watching.