Hello, and welcome to Dell Technologies Power Protect DB Series Appliance,
Avamar Data Domain Capacity Resolution Path. In this video we will talk about
space reclamation on Data Domain referenced article 192662, section 'Space Reclamation'.
This is applicable when you intend to manage the Data Domain capacity,
or when your high utilization is seen on Data Domain, or when Avamar maintenance starts
failing due to Data Domain being full and Avamar starts receiving integrity alerts due to the same.
This video was created to get familiar with Data Domain cleaning like when to go for space
reclamation, how to perform dd cleaning, how to schedule dd cleaning for it to run automatically,
details regarding dd cleaning throttle and frequency, how to check Avamar maintenance status,
how to verify checkpoints on Avamar and snapshots on DD. Now let's quickly recap on what we learned
in part one and then see details regarding space reclamation along with demo. We have seen so far
is we need to first check the utilization tier wise by logging into the Data Domain via ACM
or directly, and then check which tier is having high utilization. If the particular tier is having
a high utilization then is there a cleanable size present for that tier or not. If there
is then we will go with the space reclamation. Let's see what space reclamation talks about.
So space reclamation basically means that the DD cleaning process is the one that is responsible
for reclaiming space on Data Domain against the expired backups. If active tier utilization is
high we will initiate cleaning on the active tier and if the cloud utilization is high
we will initiate the cleaning on the cloud tier. How you are going to do that is either you can use
the command ‘filesys clean start’ for active tier and ‘cloud clean start ’ for the cloud tier
or you can also do the same through the UI. So let us see how you can do the same through UI.
First thing you need to know, here is the active tier cleaning and cloud cleaning,
there are certain default schedules that are present on Data Domain so you're going to check
if, you know, customer has reverted those schedules to never. Is it actually indeed
being present the schedules is the, are the cleaning cycles being run regularly or not?
So that can be checked once you log into the DD UI, again through ACM or directly, you will go
to the same ‘Data Management’, ‘File System’ tabs from the left panel and then go to the settings
gear icon seen at the right side. Click on it and then go to the ‘Cleaning’ tab.
Here you will see regarding active tier cleaning and cloud tier cleaning. So the schedules are
actually can be set over here, can be you know checked from here whether it is present or not.
Here we can see it is weekly, the frequency is for active tier cleaning is weekly, 6AM Tuesday
which is the default schedule, and for cloud tier the default schedule is actually for after
every four active tier cleaning. So cloud your cleaning is dependent on active tear cleaning,
after every nth number of active tear cleaning a cloud teir clean can run. Okay, so either
you can change it to two or let it be as default, but it needs to be present so that the cleaning
cycles are run automatically and there's no require, there's no need to do manual intervention
anytime. As well as you see something called ‘Throttle Percentage’ over here this is the amount
of CPU cycles that you would like to assign it to the cleaning process. 50 is the default throttle
but in instances where you would like to increase it for a certain amount of time like for
one particular cycle, like in this case you know Avamar maintenance is failing and DD is full, for
example it's 99 percent full, or you know and you want the, and you do see cleanable space, a lot of
cleanable space that means there are expired backups. There are scope of reclaiming space
and you want to give priority to the cleaning process so that it runs, you know,
in lesser duration than usual and reclaims space quickly. In that case what you do is you put your
backups on hold because there's no space on DD any which ways to accommodate any more backups,
so you put your backups on hold and increase this throttle for let's say 80 percent and you wait for
the cleaning cycle to complete. Remember whenever the cleaning cycle completes you
got to revert back the throttle back to what it was earlier. Either you can keep it to default
but don't leave it to the increased throttle because that can impact the performances
for the other process that would be running on Data Domain. Okay, so if you want uh every process
to run smoothly parallelly then do not keep this throttle, you know, at a higher level. Also
the frequency that you see over here that from our perspective and from product perspective basically
we suggest not to set it too frequently. Once in a week is default for active tier cleaning.
Once in a week is default but if you want to, you know, increase the frequency then you can
do it maximum twice a week. That should be fine but do not go for too frequent cleaning cycles
because that would impact the data locality present on Data Domain, okay and that
it causes high fragmentation and that would lead to impact on the read performance as well
so that is the reason to not have the cleaning set too frequent. That was all about cleaning
schedule throttle as well as its frequency. Okay, coming to the back to our point where
your you know utilization is high and Avamar maintenance is failing you would like to start
the, though the cleaning schedules are present you can't wait for the cleaning schedule to, you know,
until the time the next schedule run. What you would prefer in that circumstance is you would
want to initiate the cleaning cycle immediately and how you can do that from UI is you would see
an arrow over here at the bottom at the right side. Expand that here you will have an option
for ‘Active Tier Cleaning’ and a start button against it. so clicking on this the cleaning
will get started immediately. If you want to alter the throttle then please do that before
initiating the cleaning cycle and once cleaning cycle completes you can revert back the throttle
from the same place from the settings like gear icon. The same thing can be done from the
CLI as well so if you want to see the schedules you will see ‘filesys clean
show schedule’ okay you will see for cloud cleaning you will see ‘cloud clean show frequency’
uh sorry ‘frequency show’ okay. Same goes with the throttle ‘filesys clean
show throttle’ and ‘cloud clean throttle show’. If you want to set it reset it you may instead of
show command you may use set command to set it to the value required value that you want to run for,
so these are the schedule throttle and then if you want to initiate the cleaning cycles ‘filesys
clean start’ is the command for active tier cleaning cycle and ‘cloud clean start
cloud unit name’ so let's say for example cloud unit name is cloud 1 that would be your cloud
clean start command to know what cloud units are present to even run this command. For example
I have taken cloud unit name as cloud 1 here so what are the cloud unit names on your device? You
can check it through the command ‘cloud unit list’ to get the cloud unit name and then run
the ‘cloud unit cloud clean start’ command uh with the cloud unit name okay and then to check
where you know cleaning status you can do for active tier cleaning you can do ‘filesys clean
status’ and if you want to see if the cleaning is you know ongoing if you want to see the real time
progress of the cleaning you can do the watch command. This is for the active tier, same
goes for the cloud tier cleaning as well ‘cloud clean status’ and ‘cloud clean watch’ for the
real time progress and to come out of the watch commands you can do control + c.
Okay so this was all about the CLI commands that would be helpful and all of them are documented
in the KB articles that are referenced over here as well as this KB also talks about the various
phases the cleaning goes through what it does the active tear cleaning. Okay one thing to note here
is active tier cleaning as well as uh and this cloud cleaning are mutually exclusive process,
only one can run at a time they cannot run in parallel. If, let's you know, continue with the
same example as we had started so your activity utilization is high and your requirement and
preference now is to start active tear cleaning but when you check on the system even if you try
to start it you will get an error that cloud cleaning is in progress. So you can check the
cloud clean status to see what phase it is on and if you would have, if you have preference
here for example activity utilization is high you would want to start the cloud cleaning
but even though you want to start you are unable to do that since cloud cleaning is in progress. So
what you would do in that scenario is you will go ahead and stop the ongoing cloud cleaning. So how
you will do that is ‘cloud clean stop’ okay. Use this command and verify using cloud clean status,
wait for cloud cleaning to stop and reflect in the output of cloud clean status. Once that is done
then you can go ahead with ‘filesys clean start’ that is your active tier okay so always check the
status before you uh start any tier cleaning as well as remember both the tier both that
you're cleaning cannot run in parallel. This is documented as well in this reference article
as I was saying you so now once the cleaning is initiated you need to wait for it to complete
and then check whether it has been successfully completed. If DD cleaning has not successfully
completed you can contact support and we can guide you further otherwise once cleaning gets
successfully completed check whether the space is okay post cleaning or not okay. How to check space
we have already seen that in the earlier slide you check through ‘df’, our ‘filesys show space’.
Okay, now once that is done you see the space is okay or not post cleaning. If space is okay
post cleaning then you got to wait for the next successful Avamar maintenance cycle
okay and then re-run DD cleaning and why is that? So because earlier the maintenance cycle was
failing, so if there were any backups that are expired manually or if they have met retention
then that has, so that has not been communicated to DD yet. The DD doesn't have any information
since the time the earlier maintenance was failing. If there were any backups
that have been expired DD has no information about it. It only received information once Avamar sync,
uh above Avamar maintenance is done. So you need to wait for the successful
Avamar maintenance which is scheduled to run every day in the morning on Avamar
and that includes Avamar checkpoint, HFS check and GC. Okay once that is done uh you can rerun
the DD cleaning using the same commands that we have seen earlier okay, and how you can check
about this maintenance cycle whether it has been successful or not you can log into the Avamarr CLI
and run the command called ‘status.dpn’. This will show you the information about
all of those three process that are included in Avamar maintenance that is the checkpoint,
GC and HFS check okay,all of them states okay, that means the maintenance has run successfully.
You can also see the cp list command to see the check points that have been taken okay. So
Avamar contains maximum two check points over here and that should be the latest from the uh
you know maintaining cycle that has been run today okay so jan 12, so all this
this checkpoint and the rolling checkpoint has been created uh when the maintenance run today.
Okay so you've got to check if the status is okay and you have the checkpoints from
today. Once you confirm your Avamar maintenance cycle has run successfully you can go back to DD
and then you know rerun the DD cleaning. As we have seen earlier the another verification that
you can do before initiating the cleaning is you can log into the Data Domain CLI,
see the entry list, fetch the Avamar entry name and then see the snapshot against this entry
okay, and this should match with the checkpoints that you are seeing on the Avamar. So whatever
active checkpoints, active snapshots are seen on DD those should match with the checkpoints on
our rest, all should show as expired so that means Avamar has successfully communicated
to Data Domain through today's maintenance so that part of communication has been
completed and now you are good to proceed with the DD clean. You do that through filesys clean
start or cloud clean start unit cloud unit name as we have seen earlier, so
again you would wait for the cleaning to complete and check the space is okay post cleaning or not.
Using the same command ‘filesys show space’ or ‘df’ on Data Domain okay and then once
you see that it's all fine now it has reclaimed space from whatever has been expired. So far
as the Avamar maintenance has completed successfully and communicated to Data Domain so it
has taken full advantage of the cleaning this time and it has reclaimed everything it can. So that
means now it is time to acknowledge the alerts on Avamar since the maintenance has run successfully
and the maintenance ran successfully because the earlier DD cleaning was also successful
and it reclaimed some amount of space for the maintenance to go through successfully
and so we are now good to acknowledge these alerts and to acknowledge these alerts you will have to
you know contact support and we can do that for you and that will resolve your capacity concerns.