The Data Transfer Tool (or DTT) is a command line based utility that allows users to upload
or download large molecular data files to the GDC.
The tool can be downloaded from the GDC website under either "Access Data" or "Submit Data"
and is supported on Mac, Windows, and Linux systems.
Here you can also also find additional information such as the DTT User Guide, which contains
extensive information including helpful examples.
Once installed, help menus are available for both the upload and download functions within
the tool.
For example, if I wanted to see the help menu for the upload function, I would type gdc-client
upload with the -h option.
While the GDC submission portal can be used to upload all metadata entries, such as clinical
and biospecimen information, the Data Transfer Tool or the API are necessary for transferring
large files like FASTQ or BAM files to the GDC.
This tool can work with both TCP and UDP transport protocols and can resume interrupted file
transfers.
Advanced options such as specifying the chunk size or number of threads are also available.
Let's take a look at how to use the DTT for data submission.
Later I will demonstrate its use for downloading.
Use of the DTT for data upload requires that you have already set up a project and uploaded
all necessary biospecimen and experimental metadata.
You can see more about those topics in the Submission Overview and Submission portal
videos.
The data dictionary and data model pages are also excellent resources for understanding
how to provide this information to the GDC.
In this example, I have already registered my files (in other words uploaded the file
metadata) which includes the checksum and file name.
This may have been done through either the submission portal or API.
I can now use the DTT in a couple of different ways.
Either I can specify the UUID, which is a unique ID for each entity.
Alternatively, I could create a manifest, which specifies multiple files simultaneously
and passes that that information to the DTT.
Submitting data to a project in the GDC requires authentication, which is tied to a user's
dbGaP credentials.
Such authentication is provided to the DTT through use of a token, which can be downloaded
here.
Additional information on Data Access can be found on the main website under "Obtaining
Access to Controlled Data" or "Obtaining Access to Submit Data".
In this example I am providing the manifest with the -m option and token with -t
The process for using the tool for downloading files is very similar.
Let me show you how to download a controlled access file from the GDC download portal.
You may want to locate these files via the download portal interface although the API
can be used for that as well.
Perhaps I am intersted in specific files within the TCGA-BRCA project that relate to copy
number variation.
Just as with the upload function you can specify UUIDs directly or create a manifest.
You can also download a token in a similar way to the submission portal, but for open
access data a token is not required.
In this command I am downloading a single file.
In this command I am downloading a set of files specified in the manifest I downloaded.
Please see the videos on the API and Submission and Download portals for more information
on these topics.