[MUSIC PLAYING]
MISHANT KOHLI: Welcome, everyone on "What's the News?"
Today, we'll be covering our newest introduction
of gcloud storage CLI.
My name is Nishant Kohli.
I'm Outbound Product Manager.
And joining me is Sonit.
Sonit?
SONIT TAYAL: Hi, everyone.
I'm Sonit Tayal, Product Manager on Cloud Storage team.
MISHANT KOHLI: So, Sonit, as we've
discussed some of the challenges that we've
talked to our customers, you know,
when they think about moving to cloud, some of the key things
come to their mind of how do I migrate my data,
or what are the challenges that they
think about retrieving or ingesting data
into cloud storage?
There might be many tools that exist today,
but they worry about performance, parameters,
and configurations that they may think
about would be best optimal to ingest or retrieve this data.
Those things become quite daunting and challenging.
So can you help us understand what
are the things customers are doing today
and what we're doing to help them?
SONIT TAYAL: Definitely Nishant.
You point out a great problem.
Customers today have a variety of tools that they use.
They have existing CLI options, both provided by Google,
like the gsutil CLI as well as the other cloud
CLIs which are compatible with Google Cloud.
They also have tools like storage transfer service, which
allow them to take the advantage of automated transfers
to Google Cloud Storage.
But all of these tools, as you mentioned,
have the overhead of customers knowing
what are the right settings and parameters
that they need to use to get the most optimal performance.
Most often, the support cases and issues
that customers point us to are issues
related to slow transfers, and they
need help debugging and making sure
that they can get the most optimal transfer
speeds for their transfers.
So to make the transfers easy, seamless, and fast is
actually a problem that has been front and center in our minds.
And this is exactly what we've solved
with gcloud storage, which is our newest
CLI for Cloud Storage.
As the name suggests, it is gcloud storage,
so it is consistent with the rest of the cloud.
And it is fast by default. It introduces
a new hashing algorithm by using crc32 library for hashing
as opposed to the older generation, which
used the crcmod library.
So the hashing algorithm is much faster
and improves the transfer speeds greatly.
It is also great at parallel processing
because it treats every transfer problem as a graph problem,
treating every unit of work as a node,
and every dependency as an edge.
So it allows you to do a lot of work in parallel
without any overhead, again improving the speeds
of transfers greatly.
In addition to all of that, we've
also removed the overhead from the customers
to know the best parameters for their commands.
Gcloud storage detects the most optimal settings
for your transfers automatically,
without requiring you to specify any flags
or options with your command.
So this makes the command set fast
without any overhead on the customers.
MISHANT KOHLI: That all sounds great,
and I hope I don't have to worry about how the hashing algorithm
really works.
But let's talk about what are we doing to make that customer
experience much easier and better.
SONIT TAYAL: Oh, absolutely.
So we, again, have been very focused on improving
the customer experience and clarity of commands
to make it super easy for everyone to use it
in the most optimal way.
We've-- the gcloud storage command set actually improved
the clarity of commands by reducing the total number
of overall commands that we have and structuring them a lot
better in headers or units, which are much easier
to understand.
For example, everything that you need
to do for your buckets is now structured
under a header called Bucket so you have gcloud storage
buckets and everything from create to delete
falls under that set.
Similarly, for objects, you will have gcloud storage objects
and then you'll have all the commands
that you can use for interacting with objects on Cloud Storage.
MISHANT KOHLI: Now that sounds pretty easy and simple to use.
Now, for customers that are used to existing tools like gsutil,
are there anything we're doing to help them along the way
as well?
SONIT TAYAL: Definitely.
So that ease of use capability that we
have for our new customers to just adopt gcloud storage out
of the box is something which we've talked about.
But let's talk about what our existing customers that
have scripts and dependencies on gsutil can do to adopt it.
So we've introduced a translation layer
called shim which can allow your existing scripts,
which are written in gsutil, to run as gcloud storage
and get all the performance benefits of gcloud storage
without requiring you to rewrite or re-architect
any of those scripts.
So you can run your existing gsutil scripts much faster
by taking advantage of this shim.
MISHANT KOHLI: Wow, that all sounds really amazing.
So let's get started.
Can you walk us through understanding
how do I get started with this?
Where do I go?
SONIT TAYAL: Yeah.
So gcloud storage CLI is the CLI,
which is used by almost every service in Google Cloud.
So installing it is super simple.
Google Cloud ships a single SDK, which
is called the Google Cloud SDK, and you can install that SDK.
The latest version of it provides you
with the gcloud storage CLI, and installing it
is explained step-by-step in the link that we have on the slide.
You can upgrade to the SDK Version 402
when it will be available to get the gcloud storage out
of the box.
MISHANT KOHLI: Wow.
That sounds fairly simple, but I think
what would be amazing if you can walk us through a demo
and really show the true power of gcloud storage.
SONIT TAYAL: Absolutely.
Let's go into a demo where I'll show you
just how fast this gcloud storage CLI is, as
compared to gsutil.
We'll also walk through how easy it is for you to adopt and get
the benefit of shim that we've talked about,
which allows you to rerun every gsutil script that you
have as gcloud storage.
MISHANT KOHLI: Awesome.
SONIT TAYAL: So here I have a machine
with Cloud SDK installed.
You can see the version by gsutil version -l.
First, I'll be downloading a file
using the existing gsutil CLI.
So I use the gsutil -m command, which
can do multiprocessing and multithreaded downloads
for optimizing the performance of your downloads.
So here I am downloading a 1 GB file from an existing GCS
bucket to my local machine.
As you can see, the transfer is progressing
and you can see the active transfer
rate and the throughput that I'm achieving with this transfer
using gsutil.
And we have the file downloaded.
Now let's do the same transfer using gcloud storage CLI.
So for gcloud, I don't need to specify any flags.
And ultimately, the cp command can be used as-is.
Note that here I'm using the alpha version of the command,
but when we go GA with the gcloud storage CLI,
the word alpha won't be needed anymore.
So the same transfer, as you can see,
is going a lot faster with gcloud storage as opposed
to gsutil.
And we have the file downloaded.
Now let's see the shim functionality
for downloading the same file.
For this, I have to go to the boto file for my CLI.
Here I have to search for a section, which
is titled gsutil.
And once I find that section, it's
a single line that I need to add which says use_gcloud_storage
equal to true.
I save that file.
And now, I can run the same gsutil command
that I ran before.
And you would see the transfer goes
much faster because it's using the gcloud CLI underneath.
The transfer is almost complete.
And you can see the transfer rate is similar to what
we have for gcloud storage.
Hopefully, that demo gave you a good sense
of what gcloud storage can help you do.
To recap, gcloud storage CLI is the newest offering,
which makes the transfers into Google Cloud Storage
fast, without requiring you to provide any configuration
parameters.
It is also consistent with the rest of Google Cloud
because gcloud is the CLI that is used by almost
every service in Google Cloud.
In addition to all of that, we provide you
with a simple shim that can allow
you to translate all of your existing gsutil scripts
and run them to get all the performance
benefits of gcloud storage.
So you are not required to rewrite any of your scripts
but still get the performance benefits that
are offered by this new CLI.
MISHANT KOHLI: Now that sounds amazing.
And I'd like to thank everyone for joining the session today
and hope you learned something.
And we'll continue to improve your experience when using
our Cloud Storage product.
Thank you very much, and thank you, Sonit,
for joining us today.
SONIT TAYAL: Thank you.
[MUSIC PLAYING]