- [Narrator] Now let's hear
from four different data teams
about how they're handling the growing importance
of machine learning in their products and systems.
We'll be hearing from companies in health care,
energy and utilities, and software.
Our opening speakers are from
Blue Cross Blue Shield of Illinois.
They will describe how they use Emilflow
to gain a comprehensive, end to end view
of their machine learning projects.
Please welcome Derrick Higgins,
Senior Director of Data Science,
and Sonjia Waxmonsky, Senior Data Scientist
of Blue Cross Blue Shield.
- [Derrick] Hi everyone,
thanks for attending this virtual event.
I'm Derrick Higgins, Senior Director of Data Science
at Blue Cross Blue Shield of Illinois, Montana,
New Mexico, Oklahoma and Texas,
and I'm here with my colleague, Sonjia Waxmonsky,
who's a Senior Data Scientist
in our Enterprise Data Science Team,
to talk a little bit about best practices,
and not so ideal practices,
for organizing all the artifacts
that go in to a data science project.
Background of this is that Sonjia and I
have led data science projects
in a number of different organizations,
with different standards and cultures,
a lot of them large enterprises that present
particular coordination challenges.
So we have some war stories,
in terms of how things are often done,
and ways that teams can improve their practices.
So for those of you who are not super familiar
with the health insurance space,
we are one of a number of so-called blues,
health insurance companies under the
Blue Cross Blue Shield umbrella,
that operate in different areas of the United States.
We, in particular, are a not-for-profit insurer,
and the largest customer-owned insurance company
in the United States.
Of course, we are interested in controlling costs,
with the aim of keeping our member's premiums low,
but our not-for-profit status allows us to
take a broader view of our work as well,
to support the health of our members
in the communities in which we operate.
So like any data science team,
ultimately, we're tasked with solving business problems,
and some of the questions that need answering
for insurers, or payers, as we're called,
to solve our business problems,
are those that are on this slide.
For example, what can we do to improve health access,
and outcomes for local communities?
What health care provider is the best fit for a member,
given the needs that they have, their plan type,
their location, and so on;
which of our members are at high-risk
for adverse events in the near future,
and what can we do to help them avoid those,
and when requests are made for insurance reimbursement,
are they legitimate?
Are they matched to the medical need?
A lot of these questions are matched to modeling,
or analytical activities, which is where we come in.
So, for example, fraud detection is an analytical activity
that's associated with insuring
the legitimate nature of billing.
We might do modeling to estimate
a member's likelihood of being
re-admitted to the hospital after they're released.
We might build a recommendation system
to match members to providers.
That's basically our reason for being around.
Now, the first area where we've seen some challenges
in designing and maintaining
data science project repositories
is in getting a comprehensive view of the entire process,
from end to end.
So, I bet a lot of you out there have had
a conversation that goes a lot like this one.
You're working with another team,
that provides you with a data set,
and you see something weird in your data stream.
So you reach out, "Hey, can you tell me
whether something changed with a particular variable,
(mumbles), how is that being calculated?"
And they might say, "Oh, you know, we made some updates,
but we don't think we did anything
that could change your data stream."
So, you know, okay.
How do you follow up from that?
"Can you share that?
Can you tell me exactly what we've done?"
And maybe the team member who did that work
will email you, at some point,
but it's really, it's really frustrating, right?
So changes are happening that effect your work,
but there's not really an easy, or immediate way,
for you to get insight in to exactly what they are.
Similarly, you might have downstream
consumers of your work,
who embed your results and visualizations
or other systems.
So I've definitely run in to situations
where the results don't show up the way I would expect,
and it's hard to track down exactly
what that disconnect is.
Now, a major cause of this issue
of fragmentation of code and resources
is the fragmentation of work across teams
and larger organizations.
Many of you have probably heard of Conway's Law,
which says that any organization
that designs its systems, defined broadly,
will produce a design whose structure is a copy
of the organization's communication structure.
So, if your company has a data team,
and a data science team,
and a front end team and an infrastructure team
that supports all of them,
any given system you work on is likely to end up
with four different code bases,
each with it's own library of supporting resources.
And of course, this leads to lots of problems:
so bottlenecks in work that involves
coordinated changes across teams,
and an inability to reproduce processes end-to-end
and ensure that they function as intended.
And once this kind of fragmentation exists,
it can be really difficult to address,
so we want to get to a situation where
different teams do their work transparently,
and other teams have immediate insight in to it.
Where they can identify a specific version
of any code they depend on,
and ensure that they're interoperable with it.
But it's hard to decide on a single platform
or framework for sharing across teams,
because teams use different tools,
and they're always in a position
to quickly adapt to somebody else's tool change.
And of course, if there is no code
for their part of the process,
it's just a manual set of steps that somebody has to do,
then you run in to a whole different set of problems.
'Course, it would be great if all the teams involved
could just put their work in to a single place,
a single shared structure,
one shared project repository, in GitHub,
or a single shared Databricks project,
but we run in to problems there, that I just mentioned,
related to the technical proficiency
and tool chain across teams,
related to politics, and so on.
An alternative might be to have projects for each team
potentially in different repositories,
but with the possibility of linking
to specific versions across teams,
so the coupling is looser.
So it might be easier to achieve,
but of course, it still requires a certain level
of technical sophistication,
in that each team has to be using some sort of versioning.
If neither of these is viable near-term,
then the least we should try to achieve
is some sort of manually maintained links between projects.
So documentation in each repository
about any upstream dependencies, downstream consumers,
and links to the best information we have
on code locations in the containers.
With that, I will turn the floor over
to my colleague Sonjia.
- [Sonjia] Okay, so Derrick just talked about
the importance of centralizing our code
and our work product across an organization.
Now, I'm going to talk about
how we use a central repository
to store an important part of our output,
which is the analytics decisions we make,
based on our data.
So what do I mean by analytics decisions?
As data scientists, we often write code
that has these hidden configurations and parameters
embedded inside it.
This include things like model parameters,
how we filter and pull our population,
thresholds we put on scores and outputs,
and decisions we make about how we're going to
aggregate our data and return results.
Oftentimes, these parameters are not given to us nicely
by the business.
Instead, they come from research that we've done
on our own data.
For example, observing that an age
has a certain distribution,
and we set thresholds based on that.
This is what makes programming for data science
different than general software engineering.
As data scientists, we often start by
exploring our data sets in an ad hoc manner,
and understand what we have and what's missing,
and then we move on to build the code
that goes in to a reproducible-type pipeline
for scoring and modeling.
This analysis that we do can take many forms.
It can include standard modeling steps,
like corelation analysis.
We review variables for trends over time and fill rates;
we also spend time understanding
our data base data models,
or just trying to learn stories from our data
and understand more about our business
and our end users
and the problems and stories that they have.
So I've spent many years working on projects
and inheriting code written by other teams
in various formats,
and I've learned that it's really necessary
to treat this exploratory work we do up front
as a first-class work product.
This means that it should be saved and documented
and memorialized for future readers,
so that if we need to explain or debug
these embedded configurations that appear in our code,
then we have a starting point for our research.
Sometimes I will go back to my own EDA work
that I've done six months ago
and I review what I've learned,
and that will save me hours of research
understanding a problem.
So this is an example of a notebook,
written by a data scientist on one of my teams,
and he saved his correlation analysis
and gave a nice write-up about what his decisions were.
And now I have something that I can use to explain,
if I need to, why particular features were dropped.
So he saved this in a notebook,
with clear descriptions of his results.
So as Derrick mentioned,
there are multiple options for shared work environment.
GitHub supports notebook rendering,
checking notebooks in to GitHub also has the advantage
that it's pinned to your code.
The Databricks environment also supports --
has a shared notebook work space,
and this makes it very easy and quick to share
work and results with teammates.
It can be shared even when
teammates are working in different clusters.
So now, let's talk about the next phase
in the data science life cycle,
which is building the model,
and storing all the outputs and results
of that modeling process.
So this also a place where it's important
for our work to be organized,
and reproducible and transparent,
so that we're mindful of who will inherit our work
in the future.
So usually, when we start modeling, we work in a notebook.
So this is an interactive environment,
and it's easy and it allows us to debug
and review our work, and do visualizations of our results.
But the downside of our work in a notebook
is that it's only local scope.
When we exit, we only have the notebook output.
Everything that's in memory goes away.
So, we can show that we have a successful method
with our notebook,
but we don't have a model that we can deploy
and use to score future data.
So this is often what happens
when we go to productionalize a model.
So here is an example of a copy and paste
of coefficience from a logistic regression model.
So I think that everyone who's worked in insurance
has done this at least once,
and this way of productionalizing your model
is fine if you have a GLM,
but when we move to machine learning methods,
we need an alternate approach.
Specifically, we need to save
hyperparameters from our experiments,
we will need to save serialized model binaries,
we need to save our test metrics
that show we validated our model,
and we need to store other arbitrary artifacts,
like charts, for example.
An image of an RUC curve,
or additional data sets, for example,
something we want to label and analyze.
All of this, that we save,
allows another data scientist to rebuild our work
and continue where we left off when they need to.
MLflow offers a way to track all these artifacts.
We can do this with a python API
that can be called directly from our modeling pipeline.
MLflow will support storing hyperparameters,
binaries and metrics,
as well as other arbitrary artifacts.
So here is an example of a grid search
that I did in MLflow with Databricks,
and what you can see that I have here
is a way to review all of the hyperparameters
as well as metrics results from across different folds.
My results are stored in a central location
in a consistent way, so that I can review them
and others on my team can review them.
MLflow also supports a model registry
that will support model versioning,
and will show which experiment
produce a particular result.
So as with everything we've discussed,
our goal here is to store our work
in a way that's clear and transparent,
it's available to other data scientists,
and it supports people on our team,
as well as others who are going to
inherit our work in the future.
- [Derrick] Thanks, Sonjia.
So, I hope you enjoyed this survey
of some of our thoughts about particular ways
of organizing projects,
best practices and some things we've run in to
that are not so optimal.
Just so you're aware, this is a brief version
of a longer talk that we're gonna be giving
at Spark AI Summit,
so Patterns and Anti-patterns for Memorializing
Data Science Project Artifacts.
I hope this has excited your appetite,
and that you will join us for that talk as well.
Thanks.