Hello everyone and welcome. Today we will demonstrate how Serviceguard
can be integrated with SAP HANA system replication.
The solution is set up on two HPE ProLiant DL380 servers,
each with a single Virtual Machine acting as the cluster nodes. The VMs,
called nashville and brentwood, form a 2- node Serviceguard Cluster, with one node
running the primary database package and the other running the secondary
database package, which is the replicated instance of the primary database.
The packages have been configured with dependencies so that
they cannot run on the same node at the same time. The two packages were built
using the deploysappkgs command, but we have changed the package names
from the default names to make the demo easier to follow.
The screen that we will use throughout the demo is split into four quadrants.
The upper left quadrant is the SAP HANA Studio, the upper right
is Serviceguard manager. This graphical tool is used to configure and monitor
the Serviceguard SAP Cluster that we have named sapclus. The lower left quadrant
is a PuTTY window with the output of CMView CL displayed
in a loop. The lower right is a PuTTY window that we will use to input commands.
The refresh intervals have been turned down as low as possible in each of
the tools. So HANA Studio is set to 5 seconds. Serviceguard manager is set to
60 seconds and the PuTTY loop is set to 2 seconds.
The HANA Studio shows all services are started for nashville and that the HANA
System Replication status is green, with everything active and in sync.
In the CMViewCL loop, we see that the cluster is up, with nashville running
the primary package and brentwood running the secondary package.
Data is being replicated from the primary database in nashville
to the secondary database in brentwood.
In the first scenario, we test our cluster by simulating a HANA database
failure. We do this by killing the appropriate HDB Daemon.
Back at our screen, we find the process ID of the HDB Daemon
and kill it.
HANA Studio is the first system to detect the issue and reports that the database
is down.
In the CMViewCL window, the primary package is failing.
The Serviceguard takes the preemptive action of failing the secondary package
because it has a dependency on the primary package.
The secondary package helps quickly because Serviceguard leaves
the replicated database running on brentwood. When the package is down,
it goes into a relocate_wait state, waiting on a node to become available. The primary
package is now starting on brentwood and the secondary packages is able to start
on nashville. During startup of the primary package on brentwood, it detects
that the secondary database was left running, and so it initiates a takeover to promote
the secondary database to be the primary database.
Serviceguard manager reports the packages starting.
Back in the CMViewCL window, the primary package is upon brentwood
and the AUTO_RUN parameter is automatically set to disabled. In HANA Studio, we changed
the administration view to brentwood.
HANA Studio reports the database is up and running in a green state.
We need to update the refresh rate in HANA Studio from the default 60 seconds
to 5 seconds,
Serviceguard manager reports the replicated instance has started.
HANA Studio now reports that brentwood is part of a HANA System Replication pair,
but it doesn't yet know the status of the related services. The system Replication status
changes to yellow and this is better than red. This indicates that the data is not yet
in sync.
The status changes to green to indicate that the primary and secondary
instances are now in sync. Serviceguard reports the secondary package
up and running on nashville. It is now providing protection to brentwood,
the new primary node. Notice also that AUTO_RUN is still disabled for
the primary package.
At this point, Serviceguard detects that the replicated instance is in sync
and automatically changes the AUTO_RUN bit to enabled. This indicates that
the primary database is protected and now able to fail over,
even though failover is possible because the databases are in sync,
Serviceguard has disabled switching for the primary package on nashville
because it originally failed on that node. So let's reenable this with
the cmmodpkg command, before moving to the next scenario.
In scenario 1, we tested an application failure. In the second
scenario, we will show how Serviceguard reacts to a node failure. We will also
see how Serviceguard handles that same node rejoining the cluster. We have set
the AUTOSTART_CMCLD variable to 1, which allows the node
automatically rejoin the cluster after reboot.
Back at our screens, we first open vSphere
and power off the brentwood VM, where the primary package
is currently running.
Serviceguard detects brentwood is missing and reforms the cluster
into a single node cluster. The primary package goes immediately
into a relocate_wait state as a secondary package must be shut down
before the primary package can start on nashville.
The primary package is now starting on nashville and has initiated
the takeover of the secondary database.
The secondary package doesn't have an available node to run on, so it is down
in a failed state.
HANA studio has detected the database is unreachable and that the current status
can't be determined. So let's select the nashville tab
and see that some services haven't started yet. This is an indication that
the takeover isn't completed.
Serviceguard Manager has detected an issue with the cluster, and we will
determine the current status in the next refresh.
HANA Studio is updating the status of the primary database. It removed the status
of the replicated instance and popped and alert that the database is not
currently allowing connections. Primary Database is still in the takeover status.
So now nashville is reported as green and all daemons and services are running
and available. It's not showing as a HANA system replication pair
because brentwood is not available yet. That will be coming up next.
With the primary package now up on nashville, we can restart brentwood.
So we open vSphere to power on the VM.
brentwood is now booted and automatically rejoined the cluster.
The secondary package automatically starts on brentwood as well.
Serviceguard Manager is reporting the cluster is back up and running,
and it displays a warning that the AUTO_RUN bit is disabled for the primary
package on nashville. This bit was automatically changed to disabled
when the primary package detected the secondary database. When the databases
are fully in sync, the AUTO_RUN bit will automatically change to enabled.
Looking at HANA studio, it is detected that the secondary database is starting,
but it can't determine the status of all the services yet. It does show that
the two are in the process of becoming a bonded HANA system replication pair.
And now reports that the secondary database is up and running and
currently in the process of syncing with the primary database.
Serviceguard reports that both the primary and secondary packages
are up and running. The primary package still has the AUTO_RUN bit set to disabled
since the sync hasn't completed yet,
and now HANA Studio reports that both databases are in sync. So as soon as
Serviceguard detects this, it will update the AUTO_RUN bit for the primary package.
Serviceguard Manager is also reporting that the replicated instance
is up and running.
Serviceguard detects that the replication is now in sync and changes
the AUTO_RUN bit back to enabled.
We have demonstrated two different types of failures. At the database level and
at the node level. Serviceguard was able to get SAP HANA and HANA system
Replication up and running with no administrator intervention.
Serviceguard successfully managed stopping and starting of the HANA
databases and the system replication.
Thank you for taking the time to watch this demonstration of Serviceguard
integration with SAP HANA system replication.