In this presentation we’ll do a deep dive into the HPE SimpliVity Data Virtualization Platform
and talk about how data deduplication is implemented
and used within an HPE SimpliVity hyperconverged infrastructure.
An HPE SimpliVity hyperconverged infrastructure starts with an HPE ProLiant DL380 Server
running a hypervisor and one or more VMs.
HPE SimpliVity adds a data abstraction layer we call the ‘Data Virtualization Platform’.
This layer extends across all nodes in a cluster.
Within the HPE SimpliVity Data Virtualization Platform, there are a few layers with different roles.
At the top is the presentation layer.
Below that is the data management layer, where all the data is tracked within the system.
Data stores are maintained and presented in the presentation layer to make the VM files
available to the hypervisor. Representations of the virtual machine disk files
are presented within the data stores. Within the data management layer are two sub-layers:
the file system and object store. No actual data or metadata is stored
or managed in the presentation layer; only pointers down to data containers in the file system
within the data management layer, which are represented by these boxes.
Not all the data containers in the file system are VMs, however.
The boxes with a label are being presented through the presentation layer, but the rest
are backups of VMs. The data containers are effectively the same, whether they are backups
or VMs, with the only difference being whether or not they’re presented to the hypervisor.
The data containers in the file system interpret and track the metadata
and data that are deduplicated and stored in the object store.
More detail on exactly how that works later in this presentation.
But you can think of the presentation and file system layers as very thin layers
on top of the large deduplicated object store.
All the magic of the DVP really happens in the file system layer.
If we use HPE SimpliVity to clone a VM, only the data containers copied in the presentation layer
is updated to have a new VM pointing at the new data container.
None of the actual data blocks of the source VM are copied, since we already
have the unique metadata and data blocks. This HPE SimpliVity clone operation completes
and the VM appears in the vCenter inventory in a few seconds, with virtually no I/O.
The process to back up a VM is quite similar.
We make a copy of the data container, but since it’s a backup we don’t need to update
the presentation layer. Again, there is almost no I/O because we have already stored
all the unique metadata and data blocks. Only the file system needs to be updated.
A backup is important, but only because we need to do restores. When we restore from a backup
it is the same operation as an HPE SimpliVity clone. We make a copy of the data container
that is the backup and update the presentation layer to show the restored VM to the hypervisor.
Again, there is no need to copy large amounts of data as we already have the unique blocks stored.
You can see that copying, backup and restore are basically about copying the data container
and possibly updating the presentation layer if the result is a VM.
Because there is virtually no I/O and therefore no load on the disks, there is no impact
to VM performance, no noisy neighbor problems when performing backups and restore functions.
We can back up our clone VMs anytime, even several times per hour, without impacting
performance over any VMs. You no longer have to size your storage with additional overhead
to account for backups or confine them to the dark of night. Keep in mind that the backup
is a real full backup; it doesn’t have any reliance on the source VM, unlike most storage snapshots
or vSphere snapshots. Taking a backup of a VM does not alter the way its I/O is processed,
and the backup is fully independent and will continue to be viable if the source VM is deleted
or corrupted. As we expand our view even wider across the globe, you can see a very large
infrastructure that the HPE SimpliVity Data Virtualization Platform
will make much easier to manage. We call this the ‘HPE SimpliVity Federation’.
You can see here that the clusters within a federation can have a different mix of nodes.
Also notice that there is a lot of common data across all the sites that will make
data transfers very efficient. In designing the management layer of this VMware
vSphere infrastructure a customer can choose to use a single VMware vCenter server.
But this can be challenging for availability. Some customers will choose to implement
a single vCenter server per region to provide more redundancy and reduce the latency between nodes
and the management infrastructure. But the best way to provide management availability
is to implement one vCenter server per data center, which ensures that every site
can stand on its own. In any case, HPE SimpliVity nodes will continue to function
just fine without vCenter, but management of the environment will be impacted,
just like vSphere management will be. HPE has had a strong focus on simplifying hybrid IT,
and this extends beyond the standard management layers and into the higher level
management environments, like VMware vRealize Automation and HPE’s own OneView Management Suite.
By pre-creating integrations into these platforms for our customers, HPE SimpliVity is not only
easy to manage but also easy to deploy in a private cloud environment.
For management applications that don’t have pre-built integrations, HPE SimpliVity offers
a robust REST API that provides all the functionality of the vSphere Web Client plugin and more.
This can be used to create custom integrations with your own scripts or any management application
that can consume REST APIs. Now let’s dive deep.
In this first section I’ll show you how HPE SimpliVity tracks data within the
Data Virtualization Platform and how that enables the HPE SimpliVity rapid clone operation.
HPE SimpliVity not only makes the operations of data more efficient, but is also very simple
to manage. By integrating HPE SimpliVity operations with existing management platforms,
administrators no longer need to maintain or learn a separate storage or backup product.
By maintaining data in an optimized state through its entire lifecycle, data operations
like cloning can happen very fast, while creating a fully independent virtual machine.
At the core of these features is the HPE SimpliVity Data Efficiency, which is not only about
saving capacity but also about saving I/O to boost overall performance so that you never
have to process the same data more than once. Now, let’s dive in and see how we do it.
As we saw before, the Data Virtualization Platform is comprised of two main layers:
the presentation layer and the data management layer.
Within the data management layer are the file system, whose primary purpose
is tracking relationships between data and metadata, and the object store,
that actually stores both the metadata and data.
Both are architected and designed by HPE from the ground up in part of our intellectual property.
A single VM is constructed of several elements across these layers.
In the presentation layer it’s simply a pointer that connects the vSphere construct of a vmdk
to the underlying metadata in the file system. In this example VM01 contains the data
‘HYPERCONVERGENCE’ at the bottom. The other objects in the file system, with ‘M’ in them,
represent the metadata. It’s important to note that the file system doesn’t actually store
the metadata or data; it just interprets the information stored within the metadata.
This logical grouping of metadata and data is what we refer to as the ‘data container’.
Both the metadata and data are actually stored in the object store.
If we want to store the word ‘HYPERCONVERGENCE’, only one copy of each piece of data
or metadata is stored in our object store; only the unique data is written to disk.
As indicated by the flashing green rectangles, any duplicate objects are identified and no capacity
or I/O is required, as indicated by the flashing orange rectangles.
For example, E and R only have to be stored once, since they have already been stored
in the object store. We store the metadata in the object store also.
The file system isn’t storing any data; it just interprets what is in the object store.
In this way we’re actually deduplicating the metadata in addition to the physical blocks.
Metadata is stored the same as data itself. Let’s say we want to take a clone of VM01.
Using the HPE SimpliVity Rapid Clone functionality a logical copy of the data container
is created and presented to the hypervisor as ‘VM02’. Everything flashes orange.
There is no duplicate metadata or data created.
At the time it is created, the two virtual machines are exactly the same, so nothing new needs
to be stored in the object store because everything is already stored in the object store.
So we can create a full clone of the VM without incurring any I/O, even to read the metadata.
That’s how the Data Virtualization Platform manages data in the most efficient way possible.
This happens across data within the HPE SimpliVity cluster, regardless of being a VM or backup
or which logical data store the VM is provisioned to.
This makes cloning very easy and happens in just seconds. It also drastically reduces the capacity
and I/O requirements required, even if you perform frequent cloning or backup operations.
Of course, no data is completely static; writes are always occurring.
When writing data, the Data Virtualization Platform also ensures that writes occur
as efficiently as possible and that the capacity and I/O are impacted as little as possible.
When considering the relationship of metadata, an upper level piece of metadata
will uniquely identify the structure below it. If data is added or modified at the end of the tree,
then the upper level metadata must change to reflect this update.
Now let’s consider our clone by itself.
We’ll modify it by writing ‘LOVE’ in front of ‘HYPERCONVERGENCE’. We also add M8
to describe this unique set of data and connect it to the rest of the tree through M5.
Since the structure below M5 has changed to include M8 and the metadata self-describes
its child objects, this node of the structure needs to change to something else
that uniquely identifies this newly updated set of data below it. So we change M5 to M9.
M9 uniquely identifies the groupings of M8, M1 and M2.
In the same way, the structure below M7 has changed too. So it changes from M7 to M10.
This is a key component of how the Data Virtualization Platform is able to track and migrate data
so efficiently. Now we need to only store unique pieces of data and metadata,
so we’ll see only L, M8, M9 and M10 need to be added to the object store.
O, V and E already exist in the object store, so they are not written.
In this way, we can see that the upper level pieces of metadata can be used to identify
unique combinations of metadata and data that exist below them.
M7 continues to identify ‘HYPERCONVERGENCE’ and M10 is used to identify ‘LOVEHYPERCONVERGENCE’.
Using this unique approach to data virtualization, HPE SimpliVity can improve the utilization
of storage capacity, or more importantly can reduce the overall I/O, thus improving
performance across all VMs. By describing large groupings of data with metadata,
it enables our unique ability to quickly identify the underlying data.
This capability will be a key component in the next section, as we discuss how data is transported
between data centers. Moving data between data centers becomes a very lightweight operation
when data is managed like this. While both sites are deduplicated and managed independently,
the metadata is created in the same way, making it easy to compare data between sites
by sharing metadata. Using this approach, HPE SimpliVity only needs to send the truly unique
metadata and blocks of data between sites, drastically reducing the I/O normally consumed
by reading data, the bandwidth required to copy the data across the WAN, and the I/O
and capacity consumed by writing data. This means that remote backups can be very efficient,
even for the first backup of a brand new VM. The Data Virtualization Platform has the ability
to understand and share the data between sites. This is because the data efficiency is not bounded
only by the data within a single data store or any other logical boundaries.
All data within a node is in the same deduplication domain, regardless of whether
it is a virtual machine or a backup. Here we have VM02 from our previous example
in the primary site and VM03 in a remote site. Within VM03 we have written ‘LOVEHYPEORHATEIT’.
If we look at the object store on both sites, you’ll see there is duplicate data
between the two sites. We’ll push VM03 to the background so we can have more space to consider
just the VM02 remote backup. VM03 still exists in the remote site and we still have all the data
in the object store. We’ll leave the data and metadata in the object store to show how
this existing data from one VM helps to make the remote backup of a different VM more efficient.
Replication of data for a remote backup starts at the top of the tree of the data container.
So we send M10 to the remote site. Being metadata, it’s a relatively small amount
of data to transfer. It is unique, so we write it down to the object store.
Each piece of metadata describes the children objects, so every piece of metadata shared
between sites automatically identifies the next level of objects. M10 references two additional
pieces of metadata: M9 and M6. We’ll investigate each piece of metadata in turn.
M9 does not already exist in the object store of the remote site, and therefore needs to be
transferred across the WAN. It is sent to the remote site and stored in the object store,
since it is unique. Now we know the next set of objects that are children to M9,
which are M8, M1 and M2. We already have M8 and M1 in the object store, so we know
they are not unique and do not need to be sent across the WAN. In this way, we’re taking advantage
of the fact that we’re deduplicating the metadata in addition to the data.
This is where the unique power of the Data Virtualization Platform comes into play.
Since metadata is universally created from the same playbook on all sites,
we know that everything below them, in this case ‘LOVEHYPE’, is already in the object store.
So there’s no need to investigate or send those pieces of data. In other words, with two pieces
of metadata transferred across the WAN, we’ve identified two pieces of metadata
and eight pieces of actual data that do not need to be transferred.
The M8 and M1 sections could represent the entire Microsoft Windows 2012 binaries
or complete and solve Microsoft SQL Server. Since VM03 is also a Windows-based VM
and already wrote these blocks of data, the first backup of VM02 doesn’t need
to move this data across the WAN. Now we continue back at the second level.
M6 is sent and stored in the object store since it’s unique to the remote site.
Now we see that M3 and M4 are children of M6. Now we move left to right across the third level.
We already have M8 and M1, so M2 is sent over to the remote site and stored in the object store.
The remote site is now aware of the need for R, C, O and N.
Of these four pieces of data, the remote site already has R and O, so those are identified
as duplicate blocks of data. Next we move on to M3 and continue to process,
recognizing that the remote site already has a V, E and R, that are children of M3.
We do the same thing with M4, storing it in the object store and recognizing
that the remote site already has E.
Now that we’re done with the third level, we move on to the actual objects,
the first of which is C. This is the first actual piece of data that we’ve sent
across to the remote site. Keep in mind that all data is compressed inline at inception
and remains compressed through its lifecycle, including movement across the WAN.
This provides further efficiency on top of deduplication. By sending this one unique piece of data,
we can identify other places where the data is utilized. They are marked as duplicate blocks
so we won’t need to send it again. We repeat the same process for the next piece of data: N
Once again, this piece of data will be used again.
Finally, we’ll bring over G to finalize the remote backup process.
Now we have a complete copy of VM02 stored as a backup. The Data Virtualization Platform treats
this data container the same way as any other containers. The fact that there is no reference
in the presentation layer is what makes it a backup instead of a VM. By utilizing the unique
tracking of deduplicated metadata and data within the Data Virtualization Platform, we were able
to send this complete copy across the WAN by transporting only three blocks of actual data
and six pieces of metadata. These three blocks of data represent the data unique to VM02.
The rest of the data that already existed at the remote site, probably represents common data,
like Windows or SQL Server that was written by VM03. By deduplicating all the data in this way,
we were able to make even the first remote backup of VM02 more efficient.
The data deduplication platform deduplicates all data and metadata and utilizes our unique tracking
of data to create great efficiencies when moving data between sites. By creating a hierarchy
of metadata, entire sections of common data, like common OS or application files
can be quickly identified and unnecessary I/O, bandwidth and disk consumption
can be completely avoided. If common OS and application installs are written in the remote site
prior to the first remote backup, then those files will never need to traverse the WAN links.
All of this happens by simply connecting HPE SimpliVity nodes together into a single federation,
and it all happens underneath a hypervisor, with no additional technologies to set up,
configure or maintain. We’ve described how HPE SimpliVity rapid clones, data changes
and remote backups occur within the Data Virtualization Platform.
All other data operations within the HPE SimpliVity hyperconverged infrastructure
are variations on these operations. Local backups and restores are the same as rapid clones,
the only difference being that one of the data containers isn’t presented to the hypervisor
through the presentation layer. Remote restores are essentially the same as remote backups;
only the results in the presentation layer are reversed.
A VM move operation is like a remote backup; only the VM is unpresented in the primary site
and then presented in the remote site. The same efficiencies exist across all these operations.
So there’s a quick snapshot on how HPE SimpliVity uses data deduplication in order to
make data more efficient, whether we’re making copies locally or remotely.