Hello, I am Andrey Fedorov from the Imaging Data Commons team.
In a previous video, we covered the basics of the IDC portal, and how it can be used
to build cohorts from the data available in IDC.
In this video I want to give you some idea about what you can do next, once you define
a cohort.
As a reminder, Imaging Data Commons, or IDC, is a data repository within the NCI Cancer
Research Data Commons, or CRDC.
IDC connects researchers with publicly available cancer imaging data, often linked with other
types of cancer data, and co-located with cloud-based computational resources and big
data analysis tools provided by the Google Cloud Platform
Note that the functionality discussed in this video corresponds to the pilot, or pre-production
release of IDC, and may change in the subsequent releases of the platform.
Cohorts correspond to groups of cases that share certain characteristics, as defined
by the cohort filter.
To access the cohorts you saved under your IDC account, you will first need to log in,
and then navigate to the Cohorts tab of the IDC Portal.
Selecting a saved cohort will apply the cohort filter to the content of IDC, allowing you
to once again examine the attributes that define the cohort.
Once the cohort is opened, you will see the option to export the cohort manifest.
Export of the cohort manifest to a file has a number of limitations, and for most purposes
it is recommended to use the BigQuery manifest export option.
Export to BigQuery will populate a table with all the details needed to access individual
instances of the cohort, either for retrieving the corresponding files for the analysis,
or for further examination of the cohort metadata.
Note that in order to be able to export your manifest to Bigquery you will need to authenticate
with IDC using a google account!
To be able to download the cohort, make sure that â€œGCS URLâ€ column is selected
when you export the cohort manifest.
Note that inthe production release of IDC, we plan to also support access to the data
using CRDC Universal Unique Identifiers (UUID).
Remember that the BigQuery table with the exported cohort manifest is intended to serve
as a temporary location.
You will always be able to re-export the manifest from the portal, but if you want to have a
persistent manifest table, make sure youcopy it under a project you own.
With the list of GCS URLs, you can use Google Cloud gsutil to fetch the corresponding files.
You can do this from the console, or from the Jupyter or Colab notebook.
You will be able to find pointers to the example notebooks from the IDC documentation.
In order to be able to run the example notebook you will need to specify the project ID that
will be used for copying the data, and the name of the BigQuery table with the manifest
for your cohort.
Remember that charges associated with data egress out of the cloud to your computer will
be billed to the project you specify when copying the data!
Egress to a Virtual Machine on Google Cloud is free.
You can learn more about the costs associated with downloading the data from IDC storage
buckets in the IDC documentation.
If you would like to use CRDC cloud resources to analyze the data stored in IDC, you will
need to establish a billing account.
Free cloud credits are offered to support imaging research use cases that utilize Google
Cloud for the analysis of IDC-hosted data.
Please fill out the form available in IDC documentation to apply and take advantage
of this opportunity!
To learn more about IDC, please check out our documentation, and join our user forum
to ask questions.
The forum is monitored by IDC developers, who will promptly respond to your inquiries.
If you have any further questions, please visit IDC portal for the pointers to the documentation
and user forum.
Thank you for your attention, and we look forward to supporting your cancer imaging
research needs with Imaging Data Commons.