Paul Boutros: Good morning. So I will talk to you about
the results, or the initial results, of the ICGC TCGA DREAM Somatic Mutation Calling Challenge,
or I will just say Challenge. So the origin of this, at least from my end, starts off
with the idea that anytime we want to do something in cancer research what we really want to
do is translate it into the clinic. And to a clinician, often what it comes down to is
we have a number of genomic profiles that we tell them give new information about the
tumor, and then it goes into a bioinformatic black box that is supposed to generate an
answer, and then in the end it's supposed to cure cancer. And [unintelligible] fundamental
perspective that means a lot of what we have to do is give clinicians and patients confidence
in the analysis that we do.
And so the problem is, as we all know, in many cases you will have the molecular profiles,
but if we use different black boxes we might get different results. I think this, to me,
was perfectly demonstrated with a biomarker that I developed during my PhD. It was biomarker
for non-small-cell lung cancer and Richard Simon took a publically available data set
and said, "I can't validate that biomarker. It doesn't work." That's okay. Not every biomarker
is supposed to validate. But interestingly, when my post doc looked at it we were able
to replicate it, and it took round after round going back and forth with a world class statistician
to figure out where the difference was, and it turned out to be an incredibly subtle difference
that led us to have the same data set and what we thought was the same algorithm yielding
different results. It was actually a slight versioning difference in the way we used our
software.
So the only difference was the preprocessing, and yet it changed completely the conclusion
of which biomarker worked and which one didn't. We explored this in a bit more detail and
we found out that if you just change the way in which you preprocess your data -- each
of the rows is a different preprocessing, each column is a different patient -- and
you can see there is very little concordance about what we would predict is the right care
for that patient. So that of course tells us that this is going to be true in every
type of data that we look at, and it holds true for breast cancer -- in fact for breast
cancer, it's pretty funny. If you decide to just play with your algorithm you can show
that 74 percent of genes are prognostic in breast cancer; essentially every single gene,
just dependent on how you analyze your data.
And so to try to resolve this we put together -- and we is myself, Adam Margolin at Sage,
and Josh Stuart at USCS -- a challenge that would allow us to benchmark the methods for
identifying somatic mutations in cancer. So at the beginning we set out and said we only
care about accuracy. This isn't about efficiency, this isn't about the most computational or
parallelizable algorithm. We just want to know how well we can do. We started off saying
this should be only real tumor data. We backtracked on that a bit, but at the beginning we said
we'd start off with a series of then tumor/normal pairs. We chose them from two different tumor
types that vary very substantially in the cellularity -- pancreatic is very low cellularity,
prostate is quite high -- and we looked at raw unprocessed data. So we looked to make
available fast cue files, BAMs, normal aligned a couple of different ways, and all of the
clinical information or protocols that somebody might want to do -- want to know: how did
you extract the DNA, what age is the patient; those sorts of pieces of information.
Ethics approval to make a data set like that publically available turned out to be painful,
and so we organized with the ICGC DACO, an application to streamline access to the data.
Essentially there is a template that groups will fill out, get an institutional sign-off
on, and can receive ready access to the data, and we are averaging six or seven days to
get access to the Challenge data right now. We also sought and received an opinion from
the Western IRB to give confidence in many countries, which require IRB approvals, even
for the use of Challenge data like this.
But as we started to design the Challenge we also realized that there would be a big
benefit for having simulated data. A couple of reasons: one is that it might draw people
from outside the standard cancer genomics field into this work, somebody who wasn't
already developing algorithms. So the way this work is to start with a genome, be it
a cell line or a germline tumor genome, to burn in the series of SNVs and structural
variations using a tool called BAMSurgeon developed by Adam Ewing at UCSC, who is another
part of the Challenge. And then we would split this into two, take a subset of the reads,
call one the tumor, and spike in additional SNVs and SVs, again at the read level, to
create an artificial tumor/normal pair. There are five releases of these. They are coming
out every six weeks -- a third one is active right now -- and the idea is to have increasing
complexities. So the first one is relatively simple. It assumes 100 percent cellular tumor,
there are no sub-clones or anything; it is meant to give a lower bound for just how -- I'm
sorry, an upper bound for just how accurate our tools are going to be. It's quite easy
to get access to the data. You can do an application at Synapse, there is a download using GeneTorrent,
and Google has provided $200 in credits to everybody on the Cloud.
So the basic structure of the Challenge then is that we've got this human tumor data section
and the simulated tumor section, kind of challenge one and two, and we are looking at both structural
variations for the tumor data and single nucleotide variations, and both are being assessed using
balanced accuracy. And then from the simulated data we have each individual tumor, and we
have each individual tumor for structural and single nucleotide variations. So there
is a lot of sub-challenges and opportunities for people to feed back and see how the algorithms
work.
So the Challenge is being scored in different ways. For the real tumor/normal pairs we are
waiting until all of the results are available and then we are doing a random selection of
results, basically trying to balance across intersections between groups, and those are
receiving deep re-sequencing on an alternative platform. It's IonTorrent, because the initial
discovery was on Illumina. For the synthetic data sets, of course, ground truth is immediately
known. So this is the kind of timeline -- we expect to have the final results by about
November for an announcement of the winner, and will be in the validation phase with the
contest closed for challenge entries by July.
So, so far the results have been, to me, quite staggering. There have 440 entries. In fact,
this is a few days old, and synthetic 3 is closing, so we are seeing something on the
order of 10 to 15 entries a day. So these are 440 different predictions of single nucleotide
or structural variations on just three genomes. There is 270 registrants with about one or
two joining each day, and to me, maybe the nicest thing about this is that there are
ongoing post-challenge submissions. After people find that they don't do well they keep
on trying to see what happened, how can I improve my algorithm and to continue to use
these tumors as a kind of living benchmark? It's also greatly improved our ability to
simulate reads and simulate mutations, maybe eventually getting to the day where we don't
need to use real tumors; we can accurately simulate the characteristics of them.
So some preliminary results. The entries broadly reflect a single ROC curve -- that's not exactly
true, but if you kind of take a look here you can basically see a relationship between
sensitivity and specificity that reflects a ROC curve. There are some groups that are
clearly off it, but if you take a look at the sort of optimum point on the top right
corner, essentially groups are only moving out a tiny bit. There was a surprisingly large
chromosome bias. In fact, this plot even doesn't do it justice. What you will see is that the
winning algorithm and the first in silico [spelled phonetically] only wins on a subset
of the chromosomes, whereas there is significant overlap of other algorithms doing better on
one chromosome than another. Similarly, there is bias from one chromosome to another of
how accurately different groups are calling it with some chromosomes, Chromosome 11 in
particular, having very low calling rates for reasons that we haven't been able to assess
right now.
And I said this plot doesn't do it justice. In fact, the top algorithms were the least
prone to this chromosome bias. The groups that finished in sort of five through 10,
or five through 12 -- so the middle group -- showed a lot of chromosome bias in their
analyses, and it is unclear why that might have been, but they did very well on some
chromosomes, but not on others, and permutation studies show much more so than you would have
expected by chance alone. There also turned out to be, maybe unsurprisingly, significantly
different determinates of your different types of errors. So false positives were dominated
by differences in variant allele frequency and mapping quality. By contrast, mapping
quality was still important for false negatives, but normal coverage was more important. In
fact, normal coverage turned out to be more important than tumor coverage in this genome.
Now, take that with a grain of salt, because this was the 100 percent cellular genome that
we're taking here, so that may influence why we are seeing that result.
To give you a feel for it, again, each of the columns here is a different algorithm,
and these are the different importance metrics of the different things that we looked at.
So you can see, for example, variant allele frequency moving from the left, which is the
top algorithm, through to the right, turns out to be the most important determinant of
accuracy through most of the groups, and this is determined using a random forest-type [spelled
phonetically] ensemble approach using that for the importance metric. Similarly, you
can see that for most algorithms mapping quality is second most important and so forth, and
something like GC content, or for that matter homopolymer regions, were much, much less
important to determining error rates.
The other big surprise to us was surprisingly strong trinucleotide effects. So this was
the false positive rate normalized to the genome as a whole for different trinucleotide
repeats across the genome, and it's very clear that there are certain peaks. And actually
that peak is a CCG moving to a CTG, so maybe not entirely surprising, but there are several
other peaks in the genome that are of great interest. And the last thing, and maybe the
most promising thing here, is that the coding regions had substantially lower error rates
than the rest of the genome. It's almost as if we optimized our mutation callers to the
coding regions. And the top three groups were all perfect in the coding regions; then there
was a couple of recurrent false negatives, but the false negatives, the most recurrent
across groups was only 30 percent, and in fact, there were a small -- sorry, false positives.
And there's only a small number of false positives, and in fact, the first false negative only
showed up at group number 8. So in short, the vast majority of groups in the top performing
set did very, very well in the coding regions, and it actually suggests that we need to be
putting much more work into our validation of non-coding regions.
And the last take-away is that parameterization was really critical. Pretty much every single
group was able to step forward, just from having the small amount of data and feedback
that we put up on the leader board, was allowing them to almost immediately, within a day or
two, re-parameterize their algorithm, and do a much better job on calling. So in short,
having a little bit of validation per tumor allows you to do a much better job of calling,
and it might almost be a useful thing until we work out optimal parameterizations to have
as a standard part of pipelines. So in summary, we are seeing surprising trends in error profiles,
both on the big side, from chromosomal bias, to the very small side, trinucleotide bias;
and we are looking at windows of the genome and sequence contacts and things like that.
It may be that normal coverage is more important than we have given it credit for, although
take that with a grain of salt; and we are starting to identify the best methods for
mutation prediction.
For example, MuTect was the best scoring on the first two in silico data sets for SNVs,
and Delly and novoBreak, respectively, won the first two structural variant calls. And
maybe the most important thing is we hope this is starting to foment a community that
will be working on benchmarking of these algorithms and be able to carry this forward to the kind
of living benchmark that I talked about. I should thank the people on my team, who did
the pilot work, all of the challenge organizing team, especially Josh, Adam, Katie, who is
in my lab who produced much of the analysis that you have seen here, the funding partners,
and I think maybe I will close by encouraging everybody to submit. The more entries the
better we are able to look at these issues, and community participation is really what
drives this. Thank you. I appreciate your time.
[applause]
[inaudible commentary]
Paul Boutros: Yeah, we're doing it. So the question was,
will we make the calls from different callers available? Yes, actually, that's what we're
working on right now, is to be able to do that systematically for all of them, and to
start looking at ensembles to see how the error profiles of different complimentary
callers could be used together. So that's probably number two on our priority list.
Han Liang: Any other questions? So I have a question,
if submitter submit multiple -- many wrong, this will be taken into consideration when
ranking their performance? So when you -- oh.
Paul Boutros: So the question was, if somebody submits multiple
calls, would we be able to take that into account when looking at their performance?
So we do. We are not penalizing them for that in any particular way though. We only use
a subset of the data available for the leader board, and it's certainly possible to over-fit
on the leader board. In fact, we actually have seen a couple of examples of teams that
may have over-fit to the subset used for kind of providing rankings, and we think it's a
good thing to encourage teams to continually go in and improve their algorithms like that.
Thank you.
Han Liang: Thank you very much.
[applause]
Han Liang: So in TCGA we have luxury to use a fresh,
frozen sample, but in real clinical setting we don't -- often the case we only have the
paraffin-embed sample. So it's important to know what the difference between these two
settings is. So our next speaker, Dr. Erik Zmuda from Nationwide Children's Hospital,
will talk about the lessons from the genomic characterization of patient-matched frozen
and then formalin fixed paraffin-embed tissue.
[end of transcript]