SPEAKER: Hello, and welcome.
During this session, we're going to present an in-depth demo,
where we will build the necessary infrastructure
to support a fairly sophisticated scenario for EDA
workload Burst case, from customers on on-prem
to Google Cloud.
There is a lot to cover, so let's get started.
The idea behind this demonstration
is to showcase the Cloud benefits
for EDA customers that consider to start
their journey to Cloud.
Specifically, to highlight the features
such as Cloud elasticity, flexibility, and scalability,
and the real value that Cloud can bring to these companies,
which is a shorter time to market.
The ability to access much larger resource
pools in the Cloud enables engineering teams
to compress the total time it takes to complete their jobs.
And with EDA workloads running in the Cloud,
these teams can perform more design iterations per day,
and get these designs out to the market much faster.
We are going to start off by presenting this setup we've
used to create this demo.
We've tried to build a simplified architecture,
and have tried to be as close as possible to real world
EDA architectures used today by some of our customers,
and to run a flow that resembles the EDA
Burst to Cloud scenario.
This demo consists of seven steps,
and we will go over each one before diving in.
Going to start with on-prem system preparations.
From on-prem Worker, we will install and configure
the open beta, which is an open-source,
general-purpose, multi-threaded [? many ?]
core processor and framework environment.
In our case, we're exploiting it as an infrastructure that
is used to configure and run simulations and, in particular,
will serve as a source for configuration files that
will be used for regression runs in GCP.
Next, we're going to deploy to NetApp CVO instances
in GCP using Terraform, and configure
the FlexCache relationships between on-prem and GCP
environments.
We will configure a NetApp CVO1, to act as both high performance
storage for the workloads in the Cloud
and as a cache for open [INAUDIBLE] configuration files
from on-premises.
To an EDA workload in the Cloud, the configuration files
appear to be local.
And there is no need to mirror all the tools and libraries
to the Cloud.
Get just the data you need, where and when you need it.
We will configure the NetApp CVO2 instance
as a target for the Cloud regression run results.
And by creating a reverse cache relationship,
these results appear to be local to engineers
working on on-prem.
During this step, we are going to create a SLURM cluster.
Later on, we will use special arguments that
are provided to the regression run command,
and launch each test in a SLURM [INAUDIBLE],,
allowing for tests to be run in parallel.
Here, we will run two regression runs.
First we'll be using 10 GCP Workers,
and the second will use 20 GCP Workers.
The goal will be to present the effect
of compute scaling and the FlexCache technology
implementation.
Next, we're going to present one of the benefits
behind the FlexCache.
We will showcase the ability to access the same files
from both on-prem and GCP Environments
by going over the regression runs results
from both locations.
Going forward we will demonstrate a case
where this regression runs results
can be copied to Google Cloud Storage for later
on analysis or archiving.
And the last step will be the GCP Environment clean up.
We will delete it all and remain with the regression run
results in GCS.
So let's dive into the demo.
As it was presented before, we're
starting with on-prem environment configuration.
As you can see here, we're using the NetApp hardware storage
system.
From the on-prem Worker, pre-mounted
to the on-prem volume, we are installing the open pit
environment, while created configuration files
will be used as a source for the future regression runs in GCP.
We switch over to the GCP environment,
where we will deploy to NetApp CVO instances using Terraform,
followed by the FlexCache relationships configuration.
We're starting with the clear environment.
And through connection to the Bastion host,
we are verifying the correctness of [INAUDIBLE]
from configuration and starting the deployment process.
After executing the deployment, the progress
can be monitored through NetApp Cloud management interface
And once it completes, we've got the relevant information
that will be used in the next steps of the demo.
By refreshing Google Cloud console window,
we can see two new GCE instances are created.
Having completed the CVOs deployment,
we are going right in and creating the FlexCache
relationships.
First, we will create the connection
where the on-prem volume will serve as a source,
and the CVO1 will be configured with the FlexCache volume.
To speed up the configuration process,
we will use a simple pattern script
that utilizes the NetApp on-top REST API
comments to preparing the [INAUDIBLE]
to FlexCache relationship.
We insert the required information about both source
and target NetApp systems, and execute the script.
Once it's done, we are continuing
with what is called a Burst caching configuration, where
the CVO2 volume will be used as a source,
and the on-prem NetApp system will be configured
with the FlexCache volume.
We will use the same method of executing the configuration
through the prepared head-of-the-time script.
We can verify the successful configuration
by accessing the on-prem Worker and trying
to mount it to created-a minute ago on-prem FlexCache volume.
Having enabled the FlexCache relationships,
we're heading in deploying the SLURM Workload Manager.
This manager will be used to orchestrate over
the regression runs in GCP.
The link to SLURM deployment files
can be found in the comments below or in the demo
walkthrough documentation.
The SLURM will be deployed using Terraform.
So we switch over to the configuration file,
and update it with all the relevant information required
for the deployment.
Again, all the information can be found in the demo
walkthrough documentation.
It takes a couple of minutes for the installation to complete.
We can see the new SLURM instances are created
in our project and it will take additional 15 minutes
for the deployment process to complete.
While waiting for the SLURM deployment completion,
we're going to create performance dashboards,
using Google Cloud Monitoring.
The goal is to be able to follow the IO
Flow during the regression runs, and present the NetApp CVOs
read-write throughputs.
We can verify SLURM deployment's successful completion
by connecting to the SLURM Log In-Out.
It's mounted to both NetApp CVOs.
We can access on-prem data using FlexCache and SLURM
nodes are ready to be allocated to run the workload.
Now, we are ready to execute the aggression runs.
First we will be using 10 Workers
and then we'll perform the same, but scaling the Workers
from 10 to 20.
Each regression run will include 46 tests.
The run will be started from the SLUM Log In-Out
And once started, it will require the access
to relevant configuration files, which are currently located
at on-prem NetApp volume.
Once these files are cached locally in GCP,
SLURM Batch job has started, allowing for regression tests
to be run in parallel.
New regression run results directories
created at CVO2 and 10 new computing instances
are created in our project.
And now let's monitor the IO Flow.
Newly created Workers are mounting to both NetApp CVO
instances, and reading the run configuration data
through CVO1.
Since the data is not presented locally,
it is fetched from on-prem volume.
Of course, all that is completely
transparent to the workers running the test in GCP,
and the configuration files appear for them to be local.
We can see that the Worker CPU [? transition ?]
is getting higher once they're starting to run the test.
And the test results are written back
to dedicated volume created at NetApp CVO2 instance.
Once we are done with the first regression run,
we want to verify that the results
folder is accessible for both GCP and on-prem environments.
First, we are accessing the results folder located at CVO2
from GCP instance.
And as you can see, there is a list of the completed tests.
Now, using the FlexCache, we can access exactly the same folder
from on-prem Worker, and can read any of the files
from there.
Before we continue to the next step,
we can refresh the Google Cloud Console window,
and we can see that all the 10 Workers
are deleted automatically by SLURM upon job completion.
And now, we will execute the same regression run,
but this time with 20 instead of 10 Workers.
The first step will be to update the SLURM configuration
and validate it.
We're changing the number of Workers
to be deployed from 10 to 20.
[? End ?] validating, the configuration is saved.
Starting a new job right now.
The new run starts quicker, since the part
of the configuration files already cached and located
locally in the Cloud.
Once a new job is started, we can see that the new regression
run results directory is created at CVO2,
and 20 new compute instances are created at our project.
While monitoring the IO Flow, we can
identify pretty much the same behaviors
as we did with the first run.
But this time, much less data is fetched
from on-prem environment, since it's already partially cached
in the Cloud.
Workers maximum read throughput is twice higher
than with previous run.
And while writing back the results to CVO2 instance,
the overall max throughput is higher.
All that happens because we have now twice Workers running
the same workload in parallel.
This run completes much faster in comparison
to the first attempt.
The time savings are very clear.
And again, as with previous run, we can validate it.
All the results can be accessible
for the further analysis from both environments--
GCP and the on-prem.
By refreshing all Google Cloud Console window,
we can verify that all the Workers are deleted again
by the SLURM automatically upon the completion of the test.
Now, when we have all the results in the GCP,
we can move it to a less expensive storage option, where
it can be accessed later on for further analysis or just
being archived.
We're creating new GCS bucket and copying both results
directories there.
Once copied, it can be accessed through Google Cloud Console
or by any other available means.
Having done all that, and assuming that we currently
don't need anything else from our Cloud Infrastructure,
we can go ahead and clean it all.
After deleting the FlexCache configuration,
we proceed to deleting the NetApp CVOs using Terraform.
While we're running that, we can destroy SLURM deployment
as well, again, by using the Terraform.
Once everything is completed, we can
see that our GCP project is empty,
and our regression runs results are still remaining
for access through GCS.