Hi! In this demo, you see how the failover feature introduced in Oracle NoSQL Database works.
First, let me explain the usefulness of this feature.
Business critical applications have always required continuous availability.
As more organizations launch services online, for consumption by global audiences, availability and
scalability across distributed geographic regions becomes an increasingly important consideration.
To support this, Oracle recommends using availability zones. That is, it recommends you to deploy
your store across multiple zones. Since, each zone has the copy of the entire store, this
configuration provides fault isolation and business continuity. This means that when
a zone fails, write availability is automatically reestablished as long as the quorum is maintained.
Quorum here refers to the minimum number of primary nodes required in a shard to permit
electing a master for write operations. For example, if the primary nodes in a shard is
three, then two is the smallest number of nodes that represent a majority, so the quorum is two.
What happens in case a zone failure results
in a loss of quorum? In this case, the KVStore cannot automatically recover from a zone failure.
In such a scenario, an administrator can use a failover feature and manually perform a
failover procedure to recover from zone failures. Apart from disaster situations, this feature
is also useful in case of planned maintenances where you need to take one or more zones offline
that can result in a loss of quorum. Planned maintenance is actually called switchover
and the steps to perform a switchover and failover is the same.
Now, let me show you a demo of the KVStore configuration where quorum is lost and how
you can perform a failover. I have installed a 2 by 2 KVStore called ORCL using two host
machines. For the sake of this demo, I’m using the alias host1 and host2 to distinguish
the two storage nodes. Both the nodes are deployed using port 5000 and the admin services
are accessible on each node using port 5001. This 2 by 2 KVStore is deployed across two
primary zones called Mumbai and Bangalore. Let me now use the runadmin command and connect
to the store. If I run the ping command now, I get more details about the deployment.
Note that RG2-RN1 and RG1-RN2 are the master replication nodes responsible for write operations and
RG1-RN1 and RG2-RN2 are the replica replication nodes responsible for read operations in the store.
I will now try to write to the KVStore by using the put kv command and then try to
read the value by using the get command. Here, we confirm that the store is operational and
these commands run successfully. Let me exit the KV prompt.
I will now simulate a disaster scenario. I will stop the host1 storage node agent.
Now if I try to run the runadmin command on host1 and port 5000 to connect to the ORCL store,
I get an error message stating “Cannot connect to the store.” I will now exit the KV prompt.
At this point, if I switch over to the Admin Monitoring console for host1, I will see an
error message. I will use the runadmin command on host2 to connect to the ORCL store. Using
the ping command, I notice that the storage node and the replication nodes in the Mumbai
are unreachable. This is because I stopped this node’s storage node agent to simulate
a disaster situation. If I try writing to this store, I receive an error message due
to unavailability of quorum. Also, the write operation is unsuccessful.
However the read operation takes place successfully as the loss of quorum only affects the write operations.
So far we have simulated a loss of quorum scenario. Now, let us see how we can fix this
situation so that write operations can be performed on the KVStore in the existing configuration
till the issue can be resolved and the lost zone is brought back into the configuration.
To fix the lost quorum situation, you need to run two commands. First, repair the admin
quorum by specifying the zone name that is available and running in the znname parameter.
Next, enter the failover command and make the Bangalore zone primary as it is available
and the zone that failed, in our case Mumbai, offline-secondary.
After the completion of the plan, open the Admin Monitoring Console on host2.
You see the storage node in the Mumbai zone is highlighted in red to indicate that the storage node is down.
Switch to the terminal where the KV prompt is running. Enter the PING command
to see more details about the KVStore. I will now try writing to the store by entering the put kv command.
Note that the write operation is successful. I will also try reading from
the store by entering the get command. The read operation is also successful.
We have now ensured business continuity even in case of disasters that led to loss of quorum.
I will exit from the kv prompt. When the issue that caused the node to fail has been resolved,
you can bring back the storage node in that zone and start the storage node agent for
that node. Note that you have to bring back the services using the disabled-services flag.
Then, enter the runadmin command for host2 and ping the KVStore. Note that the status
of Mumbai zone is running but the nodes are still unreachable. To fix this, you need to
repair the topology. After you repair the topology, ping the KVStore again. Notice that
the Mumbai zone is up and is now a secondary zone running only replica replication nodes.
I can choose to change the Mumbai zone to primary or let it function as secondary.
To change the zone to primary, I create a clone of the current topology, change the
zone type for the Mumbai zone in the new topology, and then deploy the new topology. Using the
ping command, I can verify that the Mumbai is now running as the primary zone. I can
try the write and read commands and see that they run successfully.
In this demo, you saw how you can use the failover feature to fix a loss of quorum situation
that can occur due to zone failover or even in case of planned maintenances.
Thank you for watching this demo.