This video focuses on NCBI Clone FTP files, which can be a great help in improving genome
assemblies, detecting potential structural variation, and placing clones based on clone
end sequences.
Also see the companion video about using clone placements to interpret and improve assemblies.
A link to that video is provided at the end of this video.
Although NCBI's Clone DB web pages were recently retired, clone data will continue to be available
on the FTP site for at least one year, and possibly longer, depending on usage.
You see three top-level directories here, archive, reports and utility.
See the readme file in archive for an explanation of that directory.
The utility directory contains scripts that you can use for bulk download of FASTA sequences
for clone ends.
The reports directory is organized alphabetically by organism scientific name, and the README
files provide detailed descriptions of each file, including column identifiers.
I'll use chicken, Gallus gallus as the example for this video.
Be aware that these FTP data were updated in 2018, despite what the screenshots here
show.
The files here provide information about the sequences and genomic placements for the clones.
You find library-specific files, such as CH261 and J_AC, that begin with their clone library
abbreviations, then at the bottom are files general to the organism; Gallus gallus has
taxid 9031.
Let's review these files to see how they can help you.
We'll first look at files with information about all libraries for this organism.
At the bottom of the page, library_9031.out provides general data such as the total number
of clones with end sequences, with inserts, and that have both ends sequenced.
These numbers are given for each library and vector type.
The clone_acstate.out report provides sequence identifiers and metadata for the inserts.
FASTA for these insert sequences can be downloaded using the E-Utilities, our API for accessing
Entrez Nucleotide.
Likewise, endinfo_9031.out provides lists of sequence identifiers and metadata for the
end sequences.
To download the FASTA for these sequences, use the identifiers as input to the Perl script
in the utility directory we saw earlier.
Notice that both the clone_acstate and endinfo.out reports are available in *library*-specific
and across library versions.
The remaining files are associated with clone placements, a valuable tool for assembly assessment,
but many researchers lack the resources to generate them in-house.
Here, the data are immediately available for your use and interpretation.
Clone placements are generated by aligning end and insert sequences to genomes annotated
at NCBI and assigning the clone one or more, or possibly zero, genomic locations.
In order for a clone to receive a placement based on its end sequences, both ends must
be placed on the same assembly molecule.
All of these library-specific .gff files follow the same naming scheme.
The library abbreviation is followed by the accession.version of the assembly on which
the clones are placed.
This is followed by the Clone placement release ID, which defines the set of clone sequences
and software versions used for the placements.
And finally, the type of placements contained in the file, such as, both ends not uniquely
placed or both ends uniquely placed.
See the README_GFF.txt in the reports directory.
Note, you may find some legacy files that pre-date the use of the release id, as show
here.
We suggest always using the most recent files, unless your analysis specifically calls for
an older set of placement data.
Now let's take a closer look at how these data can help you.
I'll cover the four general tasks shown here: - Generate basic assembly QA stats
- Fill in assembly gaps - Detect mis-assembly and structural variation
- Place unlocalized or unplaced assembly molecules
And in some cases I'll use screenshots from our Genome Data Viewer to graphiclly illustrate
these tasks.
Be aware that in most cases, the task is best completed by parsing and analyzing the clone
placement files.
For the first task, generating assembly QA statistics, clone placement data can provide,
- the percentage of clones that are placed - concordant vs. discordant placements, and
- unique vs. non-unique placements
For these analyses, the clone end placement summary report, as well as the 3 clone placement
GFF files, unique_concordant, unique_discordant and multiple, are easily parsed to provide
this information.
To identify clones that contain sequence missing from an assembly, you can identify assembly
gap locations from the organism's Assembly AGP files in the /genomes/genbank FTP site.
To get these files for chicken, go to latest_assembly_versions, on to assembly_structure, Primary_Assembly,
assembled_chromosomes/, and finally, the AGP directory.
Again, use the AGP files to get assembly gap locations.
With those locations, you can parse the unique_concordant and unique_discordant placement files to identify
clones that span the gap locations.
This graphic shows that clone CH261-52K22, and several other unique_concordant placements,
spans this assembly gap in chicken chromosome 1.
Likewise, end placements near and properly oriented towards gaps may identify clones
that project into, but do not span gaps.
Use the single_end_placed files for this.
The third task is to detect regions with mis-assembly or structural variation.
These are typically revealed by analysis of discordant clone placements, that is, placements
which are larger, smaller or oriented contrary to the corresponding assembly region.
One example of using discordant placements to identify a potential region of mis-assembly
is illustrated on chicken chromosome 6.
Here we see a tight cluster, by location, of 3 discordant placements from the J_AC and
J_AD libraries in which the ends align in an inverted orientation relative to this assembly
component.
Library tracks like these present data from the unique_concordant, unique_discordant and
multiple GFF files.
This pattern suggests an issue with the orientation of this component in the assembly, consistent
with the report of an assembly error at this location to the Genome Reference Consortium
(GRC), the group that maintains the assembly for this organism.
Another example is illustrated on chicken chromosome Z.
Here you see a dramatic shift from unique concordant (blue) to mostly discordant placements,
the red clones, in three different libraries.
This is another likely region of mis-assembly.
Switching over to chromosome 21, here we see a region with both concordant and discordant
placements.
The top library is derived from the same source DNA as the assembly, but the other two libraries
come from a different source.
This heterogeneity, and the mix of concordant and discordant placements, may reflect a structurally
variant region.
Other useful files for detecting mis-assembly include the multiple and both_ends_not_uniquely_placed
files.
These can help identify genomic regions with repetitive or segmental duplications.
And the both_ends_uniquely_placed file can help detect chimerism, a hallmark of a misassembly.
The final task is to place unlocalized or unplaced assembly molecules.
Unlocalized scaffolds can be assigned to a chromosome, but can not be ordered or oriented
on that chromosome, whereas unplaced scaffolds can not be assigned to any chromosome.
Clone sequences in the both_ends_uniquely_placed GFF file can help assign these scaffolds to
chromosomal locations.
The assembly_report.txt file, which is in the same FTP location we saw earlier for the
Gallus gallus assembly_structure directory, lists the roles, that is, unlocalized or unplaced,
for all assembly sequences.
That concludes this tutorial.
If you would like help using these files, ask questions via the "Support Center" link
found on most NCBI web pages. We'd love to hear from you.