Stephen Sherry: All right, good evening. Thanks for everyone
for coming out tonight. What I'd like to do in the next five or ten minutes is just do
a brief overview of how to access the data, both on the project website and in other resources
we have either in NCBI or EBI and I'll be followed by Paul who will do a nice introduction
to the browser they've developed. So before I went into how to get the data, I just thought
I would review a little bit more of what Gabor [spelled phonetically] said. That there's
three primary data types generated by the project. We have sequence which is in a fast
queue format and as you can see here, it's, for those of you who care, it's a tag followed
by a fragment of sequence, followed by base quality scores and an Askey computer coded
system where the letters or characters are representing fred based [spelled phonetically]
quality scores. The SAM/BAM format is for the ó how the alignment's actually look.
So if you're looking in the text version of the data which is SAM, you would see a file
ó and this is a very small shot. This is just the alignment of one reed [spelled phonetically],
where you have some header information at top in red, you have alignment information,
whoops sorry óalignment information in blue here with the cigar string followed by sequence
and, again, the quality scores. You don't see this pointer up there. [inaudible]
So, these are very large files both the fast key sequences and the SAM files are now approaching
tens of terabytes of data so if you're going to download this data, there's a real investment
that you have to make in both in hardware infrastructure for disk and bandwidth at your
institution to make sure you can do this in a timely fashion. But again as Gabore and
Gil alluded to, we want to make these data useful to a broad variety of people, so we
also record the variant calls which is the summarization of the differences that pass
the QC filters. Or it could also, in raw versions, include potential variants that don't pass
the QC filters but right now for, it's just a clarity we're showing on the release files,
all of the things that look like real variants so you have a list here. This is the VCF format.
Again the links are on these slides like in Gabor's. In this format, you have chromosome
positions and identities. If there's a db stebaris number [spelled phonetically], it's
available as well as the alternate ileal [spelled phonetically] information and on files with
genotypes, you have these column delimited genotypes, one per individual. In the no genotype
version, there's only eight columns on the file and these are very compact and provide
an easy to digest, easy to parse representation of the data. Again, this is a standard format
that you can look for.
So, having reviewed those three types of data, I thought I would just take you on a quick
tour of where the data's available on the FTP site or the browser. So, thousandgenomes.org
is the ó I'm sorry I can't find my pointer again, is this a laser pointer? Ah, here.
This is the url you should start with. This is hosted by EMBL, EBI. The data behind it
is mirrored both at NCBI and EMBL and the link that's most important is probably the
data link. That's where you go to get access to all of the data. That will take you to
a new page here that reviews about the data release policy, the conditions for use for
publications, authorship and just kind of what kind of credit the consortium would appreciate
in using the data. Scrolling down the page, you'll see a review of all of the file formats
and conventions I just described and then further, how to access the data. The important
links are the two which [unintelligible] ó these two links, the ftp sites either at EBI
or NCBI are mirrored every 24 hours, so you can go to either site and you'll see the same
FTP organization and I'll just walk through that. But I'd also like to point out that
if you really are trying to mirror all of the data, that you wouldó we would strongly
encourage you to consider using the Aspera technology. This is a much faster udp-based
transmission protocol than ftp. It's about ten times faster, we get about 300 megabytes
for second if you have the infrastructure in place and it will allow you to download
ó especially if you're getting the alignments and the pasque.
If you're just doing the other files, it's not so important, the VCF files, so going
into the ftp site, just pointing out a few of the core paths that are useful to know
about, first there's read-me's up here that document, and really nice detail, kindly hosted
by Laura Clark at EBI. The organization of the ftp site and the formats of the files
ó there is a folder called data, which is where all of the alignment and sequence information
is posted by sample id, this would be the Core-ALNA [spelled phonetically] number and
for all 700, 1100 samples we have now, that's where you can get all of the data that's been
sequenced and that's updated nightly. We have pilot data which is the release sets that
have just been released into the nature paper, so if you're actually trying to find the data
sets of the VCF calls, the N-Dels [spelled phonetically], the S-Vs [spelled phonetically],
everything else that's in this pilot data folder here. Release is a folder for our past
releases, so if you're interested in historical data from 2009, 2008 or the early 2010 stuff,
that's where we put the archive releases and finally for those of you that want to be on
the bleeding edge, Gil alluded to the fact that we make publicly available all of our
work products. There's a technical folder and in this folder is where you can go and
see our latest unreleased products so these would be the work lists that the project is
working on right now ó it's caveat en tour [spelled phonetically], don't ask the help
desk why things may or may not work there because this is really the cutting edge where
the project shares the data about what we think the next release might look like. So,
it's there if you want to look at it but it's not supported as a release product until it's
in the release folder.
So, beyond the website, where else is the data available? dbSNP is the primary archive
for the snips and the N-Del data that's going to be coming out of the paper. The snips are
put in build 132 which was released a few months ago and right now dbSNP is up to 23.6
million in uniquely mapped snips. These are the single based, class one, the deletions
[spelled phonetically] from the paper will be coming out in build 133 and I believe there's
a 1.1 million N-Dels that will be loaded. And then there's other categories of things
that the project isn't really generating but are just in there. Something that we've tried
to do to help develop the VCF file format, I'm pleased to announce that we've taken all
of dbSNP and dumped it and built in VCF format. So there's a file that's here at this link
where you can go and get a map organized file of all of the content of dbSNP so it would
be everything that's uniquely placed and there is a nice, annotation in column A which is
the info column where we are annotating for every snip, the tags and there's a url at
the top, there's an excel spreadsheet that describes the 47 tags that we're using in
VCF to talk about the clinical information, so snips that are an [unintelligible] to be
clinical or diagnostic and a gene typing kit, LSDBs [spelled phonetically], they're in there,
all of the functional information if a snip is in coding reason intron [spelled phonetically],
exon [spelled phonetically], [unintelligible], 3-d structure, [unintelligible], that's all
in there.
The sequence annotation, if there are weirdnesses about how the snip aligns to a symboleze [spelled
phonetically], if it aligns to the reference but not a solera [spelled phonetically] or
vice versa there's all that kind of nuance stuff. Information about genotyping, the platforms
that surround ó genotyping platforms, if there's conflicts ó the same snip has been
genotyped at different times to the same person, it's inconsistent, we tag all that stuff in
dbSNP. So this is just a resource if you're looking for a quick way to get the entire
contents of dbSNP in one file that's the place to go. The contents of build 132 have been
put out on all our RefSeq annotations. RefSeq is an NCBI product for chromosomes mRNA proteins
and then we collaborate with EBI to make the LRGs which are the clinical sequence records,
I think, for the clinical reporting standard so variations are annotated in all of those
place.
The last thing I'd like to say is if you're trying to find sequence data by sample, we've
made some ó we have a resource called BioSample at NCBI that you can find from a pull down
menu here at the top and if you put in the tag 1000 genomes, you can do pilot one, two
or three and it will return all of the samples that were actually sequenced as that part
of the project with their Core-ALNA numbers so if you want to get ó in this example I
did pilot 2 . These are the six individuals from the two trios but likewise there's the
179 from pilot 1 and the 757 from pilot 3. You can click on any one of those samples
here and that will take you to a sample summary record which points back to dbSNP and all
the content for that person and maybe more importantly SRA to get all of the sequence
and the alignment data for a particular individual so that's kind of an easy way to jump in and
pull data by sample id.
Last slide is to say we have colleague collaborators in Amazon that have been trying to develop
cloud based computing platforms and we're happy to say that thousand genomes has been
an enthusiastic partner about putting the pilot data into the Amazon cloud. So for those
of you who are trying to pursue cloud solutions, there is an S-3 address for finding the data
and there's also an XML summary. Right now the data are chunked into five gigaBASE bins
because there is a file size limit at Amazon that I understand might be going away in the
next few months and we'll be back to whole chromosomes. So, with that I would like to
stop and maybe take a few questions and then turn it over to Paul. Thanks.
[applause]
Stephen Sherry: Oh and please speak into the mic, so the recorded
audience can hear.
Female Speaker: [unintelligible] that you would apply to the
UCF [spelled phonetically] file.
Stephen Sherry: The QC Filter is permitted for anybody who
is running a SNP calling algorithm to write their QC information, passes a default string
for things that passed a caller standard. It's not a rigorously defined column right
now.
Female Speaker: Do you have documentation for that?
Stephen Sherry: I think the methods documentation in the paper
would explain how each caller applied QC to their call sets.
Male Speaker: Hi, I downloaded the VCF from dbSNP maybe
a few weeks ago. It was labeled, I think September 30 but at least at that time it was only about
10 percent of dbSNP and I interpreted it as just the 1000 genomes related component but
you implied it's the entire dbSNP, so did it change or did I ó?
Stephen Sherry: As of 2:00PM today [laughs], ó
Male Speaker: Thank you.
Stephen Sherry: óthe full file came up, so this is a new
fresh off the press release.
[laughter]
Male Speaker: A follow up question, do I understand that
there's no more detailed ileal frequency data in this compiled dbSNP dataset that you're
describing?
Stephen Sherry: The version that we put out today does not
have ileal frequency data in it. There are ó the genotypes that we have in dbSNP are
there by population but that doesn't include all SNPs. We have ileal frequency data I think
right now on about 50 percent of dbSNPs, a lot of the legacy data didn't have ileal frequency
but we think that will get upgraded as this next set of 25 million SNPs comes in. So our
goal is to try to match dbSNP releases with the major build releases of this project and
then pull in all genotype in frequency data to keep in sync.
Male Speaker: You've been very helpful, thank you.
Stephen Sherry: Yes sir?
Male Speaker: Quite clearly this data is huge. Is there
any thought to making it available on physical media ó or for people to ship physical media
to you if you wanted to trade, sort of, bandwidth for latency?
Stephen Sherry: We can ó we're happy to talk about it ó
that's in our experience can be a lot slower and it doesn't scale well for us because it
means a lot of paralyzed conversations with hundreds of people and their drives and did
we get the right hardware to the right person. We'd encourage that the Aspera solution, if
you have the capacity to do it and if you don't than you should probably just write
info@thousandgenomes and Paul and I could try to find a solution.
Male Speaker: All right. Another question, clearly there's
a lot of a value in the pipelines that you use to analyze this data as well and given
the huge size of the data compared to the actual size of the informatics component,
is there any thought to sort of releasing a [unintelligible], something of that nature
that would provide a working environment in which the data was analyzed as well as the
raw data itself.
Stephen Sherry: I think there are some groups that are trying
to consider putting their pipeline methods into the cloud and so it could be run and
like an Amazon environment or something. I know, just from our personal experience at
NCBI, there is a lot of infrastructure optimization that had to go into get this pipeline to run
and I don't know if an image would translate effectively. It's really ó I mean the performance
of this is tied to the hardware characteristics of your server cluster and your compute farm
and we'd have to explore that. So, I think each method developer has their own goals
about how to release their product and I'd just say that they've been encouraged to do
that ó I think technology transfer is one of the goals of the project. We have a pipeline
at NCBI, Sanger [spelled phonetically] has a pipeline, Broad [spelled phonetically] has
a pipeline, so I think you could maybe talk individually to them. The project won't speak
for the pipelines as a whole. Anything else? All right, then I'd like to introduce Paul
Flicek, my colleague at EBI.