So, I'm sure all of you guys are familiar with the idea that we require a minimum
of three nodes to form a Nutanix cluster. Now, if you have been following our
recent features, we had come up with a single-node cluster in our previous
major release, AOS 5.5. With our latest 5.6 release, we bring to you
two-node clusters. I recommend you watch a single-node cluster video first. If you
haven't done it already, you should see the link below.
Hi guys, welcome to another video of Tech TopX. Quite similar to the single-node cluster, the
two-node clusters are also designed for ROBO implementations that require a
lower cost and yet higher resiliency. However, it provides more of the
resiliency features of a three-node cluster that you're more familiar with:
RF2 for data is spanned across two nodes, meaning the cluster can tolerate a
node failure. During an AOS upgrade or a hypervisor upgrade, the user VMs will
still be functional. The key difference here is that there is an external
Witness VM that must be running in a separate failure domain and serves the
purpose of a tiebreaker. So let's go to the architecture of it which will clarify this.
So, this is a two-node cluster with A and B being the two nodes and here is a witness VM.
Now, Genesis on both the nodes sends heartbeat messages every two seconds
and both the nodes contact the Witness VM at 60 seconds apart. And this
is a normal state. The way it works is, in case of a failure, like a network
disconnect, or a node failure and Genesis times out for more than ten seconds,
the two-node cluster transitions to a single-node state. So, let's say there is
a network disconnect
and Genesis times out.
So now what happens here is that, both the nodes, A and B, requests for leadership
from the Fitness VM and the witness picks one of the two nodes as a leader.
So, let's assume A becomes a leader here.
So now, there are few steps that follow this. Now since A is the leader, A is
going to initiate the transition to a single-node mode. So what happens here is
that Zookeeper restarts in standalone mode.
And both node A and node B catch
that node A is a leader and node B is going to shut down all of its services
except Genesis. Genesis is going to update Zookeeper that the cluster is
going to a standalone state. Following this, Cassandra sees this update and
detaches node B from the metadata ring. Once the metadata has all been migrated
to node A, next curator would migrate all the actual data to the local disks in node A.
And then the transition to a single node is complete. So, let's say the
network connectivity has been restored. Now Genesis on both nodes A and B can
talk to each other. So, essentially what happens is that the cluster transitions
back to a two-node state. Now, the transition back is pretty slow because
we want to ensure that the heartbeats are stable.
Usually it takes up to 15 minutes and the process is pretty much the reverse
of the failure scenario. Zookeeper restarts and we update Zookeeper with
the intent to switch back to the two-node state.
Node B gets added to the
metadata ring and Cassandra migrates some of the metadata from node A to node B.
And now, the actual data starts getting replicated from the disks on node A to
the disks on node B. Curator takes care of this.
And finally, when all the services have completed, the leader is removed.
And this brings it back to the normal state.
Now let's jump over to a quick demo of how this is done.
This is Prism, with the same interface that you are familiar with.
I'm trying to login.
Notice that right away it asks you to configure the Witness VM. Now, I have
already created a witness VM here.
So as you can see, there are just two hosts in this cluster.
And this is the Witness VM configuration. You can see that this cluster has been
registered now. You can use the same Witness VM as in Metro, if you use that
feature, or it can be downloaded from the support portal. Now jumping over to the
command line of node A - this is in the normal state now.
As you can see the metadata is equally distributed between the two nodes. Now I'm going to simulate
a failure scenario by creating a network disconnect between the two nodes.
This is node B.
Now with this command I'm going to add a delay of 12 seconds for all the
packets going out of eth0 on node B. Now there's going to be a delay of more
than 10 seconds between node A and node B, and Genesis is going to timeout.
Now as you can see, in the Witness page, there is an update that one node is down.
Which is node B in this case, and node A has been picked as a leader. So, now the
transition to single-node mode can take 30 to 60 seconds. The client VMs
can experience IO timeouts during this period and Prism will also be
inaccessible for a bit. Jumping back to node A.
If you look at the transition change here, you see that a leader has been
picked, which is node A in this case. And, the cluster is operating in standalone
mode right now. If I run the nodetool command again,
I see that node B has been
removed from the ring and node A owns 100% of the metadata. Now, I'm going to reset the
delay in node B. So now, the cluster will transition from the single-node state
back to the two-node cluster. As I mentioned before, this transition will
take a longer time as we want to ensure the heartbeats between the two nodes
are stable. And, if it's been stable for 15 minutes, the fail back is initiated.
Now that we have waited for a sufficient time, let's go check the Zookeeper configuration.
You can see the transition changes here. This was what we saw before,
when it transitioned to the standalone mode. And now you can see that it has
switched back to the two-node state and back to a normal state, here. You can see
that node B has been added back to the ring and the metadata is equally
distributed between both the nodes.
You can see all the services on both the nodes are up. Please note that it'll take
significant time for the data to get replicated on two node B's disks. So
once that is done, the leader will be reset and you should see the update on
the Witness VM. So that is the demo of how the feature works.
Like the single-node cluster, we do have a few limitations. This is supported only on
the NX-1175 G5 platforms today. Only AHV and ESX are the hypervisors
supported. You will not be able to expand the cluster to make it a three-node
cluster. Also, features like dedupe, erasure coding, and AFS are not supported today.
So that's all for this episode. Check out the other videos on our 5.6
release. As always, be sure to subscribe and leave a comment.
Thank you for watching and have a great day.