NCBI provides free and open access to publicly available sequence data for all taxa in GenBank.
However, we know that finding and retrieving custom datasets has typically required specialized
knowledge of NCBI database structure and query syntax.
To help alleviate this problem NCBI has developed Datasets. This new resource supports intuitive
and flexible access to genomes and genes for a broad range of taxa. We will introduce
Datasets and provide a tutorial for both web and programmatic entry points.
This Datasets home page provides a description of the project and quick entry points to all
data and services. Notice that this resource is in beta and as
we develop to add new features and functionality, we want your feedback.
Now let’s use the web interface to download a custom genome dataset.
In this example I'll show how you can easily find the latest sequence and annotation for
the human reference genome, GRCh38.
In the ‘Browsing genome datasets’ section we have links to popular organisms, including
human.
Click on human to go to a table of all the latest RefSeq and GenBank genome assemblies
for human. The table shows you all available assemblies. The latest RefSeq reference genome
is at the top. Within the row are some basic stats such as genome size, contig N50, the
latest annotation release number.
To download the data, select the assemblies you want and click download.
The dialog box allows you to select genome, protein and transcript FASTA sequence and
a choice of annotation types including GTF, GFF3, and GBFF, GenBank flat file format.
Your dataset is delivered in a structured zip file that contains all your selected files
in a single 'data' folder.
The data report includes detailed information on the assembly, such as the assembly name,
BioProject and statistics like the contig N50, sequence length and number of scaffolds.
You’ll also find summary statistics on the annotation
If you are looking for a non-reference human genome dataset you can browse the table.
Or if you want a dataset from another organism, use the organism filter at the top of the
table to search by scientific or common name. And you can browse the tree to see which groups
have assembled genomes. You can also filter by RefSeq to get the high
quality genomes annotated by NCBI.
If your task requires a much larger dataset, for example, if you want to build a human
or primate pangenome, you will need to download sequence, annotation and metadata for several
assemblies. This task is better handled through programmatic access and more advanced filtering
options. Datasets offers both command line and API access.
To demonstrate how to use Datasets command line tools, I’ll start with a simple example
that shows how to gather sequence and metadata for a single chromosome across several human
genome assemblies.
On the Datasets homepage, select 'Command line' and you’ll see the download instructions.
For this demo I’ll be using a Linux terminal. To download the datasets program, you could
simply copy and paste this curl command, or copy the URL and paste into your own curl
command.
Use datasets –-help to see the datasets command options.
You can retrieve metadata on all human genome assemblies using the assembly-descriptors
command. The taxon command takes scientific or common names, as well as NCBI Taxonomy
IDs. Or use the accession command to specify an Assembly accession number.
The assembly information is returned in JSON format. But you may want to compare just a
few fields out of this file, so use your favorite json parser; here we are using jq.
For example, this command retrieves assembly accession, assembly level, and assembly name,
and outputs the data in tsv format. {jq -r '.assemblies[] | [.assembly.assembly_accession,
.assembly.assembly_level, .assembly.display_name] | @ tsv ' human.json > human.tsv}
This grep command shows you the assemblies for which datasets downloaded chromosome information.
Now we are going to customize the download. I'll first put the assembly accessions into
a file then use the command, datasets download assembly, and retrieve only chromosome 21
for our list of assemblies.
If you are an application developer, we have a public API you can use. Detailed documentation
is available on GitHub, and we encourage you to explore our Python library and corresponding
Jupyter notebooks.
Okay, we've shown you how to get large genome datasets. Let's see how to access gene data
and annotation, including sequence data for the gene, its transcripts, and proteins.
Let's say you want to obtain the latest sequence and metadata on SARS-CoV-2 host genes. A recent
paper in Nature identified 332 host proteins that interact with SARS-CoV-2 proteins. We'll
get the latest sequence and metadata for a subset of those genes.
Starting from the Datasets homepage, gene data can be accessed through the ‘Data tables’
link. To retrieve gene datasets, you can upload
a text file with a list of the NCBI GeneIDs or gene symbols, or
manually enter a list of GeneIDs or symbols, plus an organism name.
Say you’re interested in learning more about the human host proteins that interact with
the viral envelope protein. You can create a gene table based on the human
gene symbols taken from the paper. {CWC27, ZC3H18, BRD4, SLC44A2, BRD2, AP3B1}
Customize your columns to get the information you want about each gene, including genomic
coordinates, and resources outside of NCBI, like UniProt accession.
If you want more information about the proteins encoded by a particular gene in the table,
in the Proteins column, click the circled number to switch to a transcript & protein
view. For BRD4, this shows you the transcript and
protein lengths and it associates the accessions with the protein isoforms.
Going back to our six Envelope gene interacting proteins,
I've selected just the BRD4 and BRD2 genes, and will download a dataset containing the
information in the table, plus the sequence and annotation for those two genes.
The dataset is downloaded as a zip file that contains gene, RNA and protein sequences,
and metadata and annotation described in two files, the data_report and the data__table.
The data_table is tab-delimited and opens readily in Excel, if you want that.
And the data_report file contains annotation, including the genomic ranges for all exons.
The files for genomic, transcript and protein sequence have informative FASTA headers that
allow unambiguous identification of the data.
Earlier in the video we showed you the command-line download of genomes datasets.
Let's now use the command-line to download gene datasets.
Currently, you can retrieve gene data by three different query inputs. GeneID, gene symbol,
or accession. Accession here refers to either a transcript or protein accession number.
For organisms other than human, gene symbol must be combined with a species name; use
the taxon flag.
In our example, we'll input a text file with a list of gene symbols.
Here our input is a list of SARS-CoV-2 host genes.
The command to retrieve gene data is very intuitive, datasets download gene symbol,
and you retrieve a structured zip file containing sequence and metadata including,
gene fasta, transcript fasta, protein fasta and metadata in either tsv or yaml format.
Note that we plan to move to the json lines format for the data_report file.
The data table contains a range of gene information for the gene, the transcripts, and the proteins.
As well as additional data, such as the external identifiers for SwissProt and EnsemblGeneIDs.
Again, with standard unix tools you can parse the available data fields to build a custom
table of the gene information you need.
As we have demonstrated, the default download includes all available sequence files. If
you want to filter you download package to a particular sequence type or maybe you just
want the metadata, we provide flags to exclude specific file types.
In this example we’re excluding the transcript and protein files, retrieving just the gene
fasta sequence and metadata.
You can also use basic unix tools to print the fasta headers for gene sequence, for example.
Finally, we mentioned earlier the public API for genome datasets. The API can also be used
for gene datasets. We have corresponding Jupyter notebooks to
help with learning the gene-specific parts of the python library.
That concludes this demonstration. Thank you for attending our Datasets CoLab.
We'll be having a live Q&A session at 2:30 Eastern on October 29; please join us.
And anytime after the meeting, please let us know how we can improve the Datasets experience
to better meet your needs. You can use the Feedback button the the Datasets
web pages, or write to: info@ncbi.nlm.nih.gov