You can now access all of the Sequence Read Archive's vast public and controlled-access
human datasets in the cloud.
This transforms a laptop into a high performance computing device and can ease struggles with
speed and storage issues you might encounter using FTP to download large datasets from
the NCBI over FTP.
This short video demonstrates the steps needed to access public SRA data on Google Cloud
Platform, or GCP.
At the end of the video you can link to a companion video describing access through
Amazon Web Services.
I'll cover these three activities: 1) Setting up a new cloud instance
2) Downloading and installing the SRA Toolkit within your instance
3) Configuring and testing the Toolkit installation.
To set up a new cloud instance, start at cloud.google.com, and if you are new to GCP, first go to the
Docs link in the upper right of the page.
And if you just want to try out accessing SRA data, we suggest using the GCP free tier;
you can read about how that works here.
I've now signed in to my account, and I've gone the console.
To start a new instance, use the Navigation menu on the upper left to go to Compute Engine,
VM instances, and click Create.
I'll give my instance a name, and for GCP, as long as the Region and Zone begin with
us-, you'll be fine.
At the bottom of the page, click Create.
I've now have my instance running.
You will typically interact with your instance via ssh, including for our next steps, downloading
and installing the SRA Toolkit.
The steps for downloading the Toolkit are nicely laid out in our github wiki at ncbi/sra-tools,
including installing and configuration.
I am not going to show the download and install, but be sure to get the latest version, 2.10.3
or above.
I've installed that version, and I will show configuring the toolkit, because there are
a few critical steps required to access these public data in the cloud.
Type vdb-config -i to get an interactive display for the configuration.
The first requirement is to enable remote access, which should be selected by default.
You can navigate this config menu either by tab or underlined letter.
Type c, and you can decide whether you want data cached locally or not, and where you
want it cached.
If local cache is enabled, be sure that you specify a directory with sufficient disk space.
The next two settings are critical.
Type g to set GCP as the cloud provider, then type r to set report cloud instance identity.
That's it for configuring access.
I'll type s then o to save, and x to exit.
If you want to review these steps in more detail, see section 05 Toolkit Configuration
on the github wiki page.
Let's run a quick test of our toolkit installation.
Type fasterq-dump and an SRR accession (SRR5368359)
This run takes several seconds to retrieve, And there are the two fastq files.
That's all for this demo.
We’d love to have your feedback and questions.
For help with SRA data and the toolkit, write to sra@ncbi.nlm.nih.gov.
For feedback and other questions, please write to suggest@ncbi.nlm.nih.gov