LYNN LANGIT: Hi, I'm Lynn Langit, cloud architect
and developer and the first Google Developer Expert,
or GDE.
Lately, I've been working with bioinformatics clients
building cloud-based genomic-scale pipelines.
In this talk, I'm going to talk about patterns
for implementing effective pipelines on the Google Cloud
Platform.
There are four parts, what is needed
in this area, why do we need to have patterns
and how do you learn them, how are
you going to use pattern information,
and how can you learn more.
First we need to understand the scale and scope of the problem.
In my work as a cloud architect, I've
worked with some pretty big data sets
in fintech in New York and London, in adtech in LA,
but genomic-scale data sets are a whole new level.
The amount of data that's being generated
by mostly human genomic analysis requires new patterns
for building effective analysis pipelines in the cloud.
Now, when I talk about scale, what do I mean?
Can I give an example?
Yeah, I will.
A recent client that I've been working with
is the Broad Institute at MIT and Harvard.
On average, they've been putting in 17 terabytes per day
into the Google Cloud.
This is an astounding number.
Think about it in your own work in data pipelining.
Are any of you working at anywhere
close to the scale yet?
If not, I think you will be, even if you're not in genomics.
And the reason for this is the amount
of data that's being generated.
I've had other projects in the IoT area in adtech,
and they come close to this.
But genomics is at the cutting edge.
So what are the challenges in building effective GCP
pipelines at this scale?
Well, there is a lot of them.
First, cloud practices-- which services
should our researchers use?
Should they use virtual machines?
Should they use BigQuery?
Should they use higher level services?
How are they going to understand DevOps practices?
How are they going to work with configuration as code?
And really importantly, how are they
going to learn about service cost control?
You can make a very expensive mistake very fast
with this amount of data.
Speaking of data, how are we going
to try out and learn these services?
The data that many of my clients work with
is personally identifiable human health data.
How are we going to work with downsampled data sets?
These data sets are huge.
They have 3 billion letters.
And they do analysis across the genome.
Speaking of the genome, researchers really
want to work with genomics examples.
Most of the examples in Learning Cloud Services
are generic, hello world.
And that really doesn't resonate my customers.
On top of that, in genomics, there
is an increasing approach moving towards tools and libraries
such as GATK, the Genomic Analysis Tool
Kit, and the "widdle," or WDL, Workflow Definition Language,
the next Nextflow library and language, NF,
and Galaxy Project.
There are many others.
So what I've done to address this knowledge gap is,
I've created an open-source course on GitHub.
It is 100% free, because it's on GitHub.
And it consists of notes marked on pages, screencasts, code
examples, and sample data.
Importantly, all of the examples use genomic data.
It's called GCP for Bioinformatics.
So this course introduces researchers
to working with raw Google Cloud infrastructure
or systems for genomic analysis built on the Google Cloud,
including Tera.bio from the Broad Institute in [INAUDIBLE],,
VarianceSpark Library from CSIRO Bioinformatics in Sydney,
Australia, Nextflow.Io from the group in Europe,
Galaxy Project, which is global, and others.
Now, to get an understanding of the complexity,
let's first consider batch compute for analysis.
If you start at the bottom of the diagram,
although today we're talking about the Google Cloud,
many of my researchers are faced with the additional complexity
of collaborating with researchers who work
with a different vendor cloud.
So we have a number of public vendor clouds.
On top of that, they'll build their analysis
on clusters of virtual machines, and increasingly, containers.
And on top of that, many of them are now
starting to use orchestration layers, such as Spark
or Kubernetes.
On top of all that cloud infrastructure,
researchers are often using these bioinformatics tools
and libraries that I previously mentioned,
such as Tera.bio, Nextflow, Galaxy, and more.
Those libraries have associated languages,
such as "widdle," WDL, CWL, Common Workflow
Language, Nextflow, NF, or GA.
And those tools often have orchestration layers,
such as Cromwell for Tera.bio or MiniWDL.
Pipeline's API is actually a Google API
that works in conjunction with a lot
of these higher-level libraries.
Does this look confusing?
Well, it is.
There is a lot of different tools and libraries
to understand.
It's one of the reasons that I made this course,
to break all these services into parts and pieces
so that they can be learned and understood individually
and then combined effectively.
Now, in addition to batch compute,
you'll notice a similar but non-identical kind of pattern
here.
And this is interactive analysis.
Now, this keys around the center square,
in the center of the center square, the iPython Notebook,
or ipymb.
It is becoming a pattern in genomics
to create reproducible research in the form of a Jupyter
Notebook.
And so an important pattern in pipelining
is scaling those notebook environments
using Jupyter Hub or Google Colabs.
So again, there are a number of parts and pieces
to working with these pipelining infrastructures.
And my course covers all of these on the Google Cloud.
So drilling into the highlights of what my course covers,
it really is designed to be consumed in the way
that you actually work with the cloud.
There is a level zero for setting up your account
and setting up cost control.
Level one is files and data, because, of course,
we're computing on files.
And I cover core services, such as storage buckets, Google
Cloud Storage.
I cover public data sets that are available,
such as 1,000 Genomes and other data
sets for genomics that have restricted
access, services like BigQuery, which
provides serverless SQL querying on top of files
and can be used for genomic type files, such as VCF, or Variant
Calling Format.
Also, I cover higher level services, such as Tera.bio
reference data sets.
In the compute area, I cover topics
such as virtual machines, running containers,
and of course, clusters of each of these,
and also include information about serverless or functions
at the workflow level, but also the Google life
sciences, API which is called the Pipelines API, Nextflow,
and higher level services such as Tera.bio.
Additionally, to cover the interactive pattern,
I have information about working with and setting up
Jupyter Notebook environments on the Google Cloud platform.
I also include modern DevOps practices such as CI/CD,
or Continuous Integration and Continuous Deployment.
And I have information about machine learning
that's specific to genomics, covering examples
of libraries like Deep Variant, TensorFlow IO, and Google
Nucleus.
Another aspect of the course design
is that my examples are designed to be quick.
One of the challenges that I found
when I was learning genomics tools as a cloud architect is
many of the examples reflected real world.
And real world can be huge amounts of data.
So a hello world can take minutes, or hours, or even
days to run.
And that just wasn't a way that I wanted to work.
So I provide example data.
And I provide specific steps to try out each service.
My course is designed to run on a trial Google Cloud Platform
account.
And if you run with my example data sets,
it should be a nearly no cost.
I never pay more than $1 or $2.
And importantly, most examples run in five minutes or less.
So you can quickly get up and running.
And you can find out if the service
is going to work for your particular analysis needs.
Another aspect to the way I've designed the course is I've
designed it at three levels for each of the service areas,
so for example, for files, for data, for compute,
so on and so forth.
So the three levels are infrastructure as a service.
And this is for researchers who want to set up, control,
and manage everything.
The level of data, for example, they
might want to set up their own RDBMS, or relational database.
They might want to set up their own NoSQL environment
on virtual machines or on Google Cloud Storage buckets.
At the compute layer, these researchers
would prefer to set up their own clusters of Google Compute
Engine virtual machines or Docker images on Google
Kubernetes Engine.
There are other researchers who would prefer
to work at the platform level.
They would rather run their analysis over managed services
and let Google handle more of the underlying infrastructure.
So data, this looks like data lakes
that are running on GCS buckets, Google Cloud Storage
buckets, and use of public reference genomics data.
At the compute layer, this often manifests
by using services such as Dataproc or the Google Life
Sciences API, which efficiently handles orchestration of groups
of virtual machines, allowing the researchers to dynamically
spin up and spin down compute resources.
And it allows them to take advantage
of aspects that are specific to Google Cloud,
such as preemptable instances, so that they
can save lots of money, which is really important at this scale.
Still, other researchers prefer to work at the software
as a service level.
And one example of this is using the Tera.bio system,
which is a web UI that was designed
by the Broad institute in [INAUDIBLE] that currently runs
on top of the Google Cloud Platform.
In this case, rather than interacting with the GCP
console or using tools such as gsutil,
my researchers will go to the Tera.bio platform.
And they can access both their own data
and publicly available genomic data through the Tera UI.
Also they can spin up and spin down Jupyter Notebook instances
and compute clusters that run on top of the Life Sciences API
through the web UI.
Now, that's a lot of different areas to think about.
So I think at this point, it will be helpful
if we actually take just a brief look at the course on GitHub.
In this demo, I want to show you my open-source course that I
built, GCP for Bioinformatics.
So first, we're going to go out to the site on GitHub.
And this is where all the artifacts for the course
are located.
You can see that the course is organized by topic area.
And it's set up in folders.
So they're numbered in a logical order.
You would, of course, start by setting up your GCP account.
Then you would work with files of data.
Then you would work with virtual machines, Docker containers,
or functions, so compute, and multiple instances.
You may add machine learning.
You also might use code and cloud service
tools, such as continuous integration
and continuous deployment, container registries,
and other tools.
In each section, the organization
is set up as follows.
At the beginning of each section, there is a read me.
So there is a main read me.
I'll just scroll down here.
And you can see that I've consistently
used Emoji throughout the repository
to make it easier to scan.
You can see the list here.
As it says, this repo includes content you
can read, watch, or run.
So you can read pages of the repo.
You can think of them as notes.
You can watch linked YouTube screencasts.
They're demonstrations.
You can try out the computational examples
using Jupyter Notebooks.
There are linked GitHub repos.
And there are additional advanced topic links
at the end of each section.
On each page, there will be a linked YouTube video.
So you just click the graphic to go to the video.
If you prefer to learn by watching screencasts,
you can simply go to the playlists
that I've created and linked.
And as you can see, there are 27 different short screencasts,
mostly around five minutes-- some of them are six or seven--
that cover the topics via demonstrations.
Now, just to get an idea of how each topic is set up,
we'll take a look at one topic area
just so you can understand the structure, which
is consistent throughout the course.
So here we have in the files and data section--
in that section, we have a topic called SQL questions.
And this topic is covering using the BigQuery service
with a bioinformatics or genomic example.
So you can see, first, there is a little explanation about it.
And then these notes pages are very
action-oriented in the design, in that they're
created so that you can actually try out
the service with a genomics example within 5 to 10 minutes,
and get a result that quickly as well.
So this is a little bit unusual.
And that's one of the reasons I made the course,
because I've seen a lot of bioinformatics examples
that take hours to run.
And I really, in course design, wanted
to create examples that would cost basically nothing
to try out because of the tiny sizes of the sample data sets
and would be just really quick and easy.
Now, in the case of BigQuery, I've
included a number of screenshots in this notes page,
because I'm assuming that, for most people taking the course,
this will be a new service.
So you can see, for example, I've used this screenshot
in the beginning of this page.
Now, this page actually is a link out
to an additional course.
And this is a course on using the ANSI SQL
language against a sample genomic data set.
And if you just go down the page,
this page is a little bit longer than some of the other pages,
because it's a three-part course.
And you can see, for example, we've got SQL queries here
that you can actually run.
Now, just to show you how this is intended to be used,
the course is designed to be used with a trial GCP
actual account so you can run the lessons and learn by doing.
So for this particular one, if I scroll
way down here for a second--
and I'll just stop here.
This is an example using a concept called a self-join
on a very, very small, but a domain-specific example.
And so the idea here is to return a hierarchy.
And the lesson, as you can see, is
write a SQL query to return the grandparent category
of glycine binding.
So to show you the environment where you'd work with this,
this is BigQuery in GCP.
And for this course, I've created a publicly available
really small sample data set.
So you can see here, I'll click on this experiments table.
And I can actually preview it.
And it has just five rows, again,
really tiny examples here.
One of the concepts, as I've mentioned,
in talking about this course is that you
can explore these services with examples that come
from genomics that are tiny.
So they can do it basically at no cost
and in just a short amount of time.
So the course is completely on GitHub.
So it's completely free.
It's completely open source.
I really welcome any suggestions from the community.
The best way to do that is through a poll request, sort
of the typical GitHub process.
And I really look forward to the contributions
and the feedback of the bioinformatics research
community.
It's Lynn Langit.
And this course is GCP for Bioinformatics.
Thank you.