Welcome back to "This is My Architecture." Today, I'm joined by Bilal Bashir from Healthfirst.
Hey Bilal. - Hey.
Well, tell us a little bit about what Healthfirst does.
Healthfirst is the largest, nonprofit health insurance plan in downstate New York City.
We are proud to serve over 1.4 million members, predominantly in the medicaid and medicare space.
That's great. So, I love your title, you know, you're the Senior Data Wrangler,
so you're swimming in data all day. That's right.
And, one of the things that we like to do is revisit an architecture.
And you're going to come and tell us about how you've improved it, since we last spoke to you.
But, tell us a little bit about what you do with data for your members.
So, we utilize the information from our claims information, our member information,
and we utilize that to make it better for our member.
Our digital strategy is Enterprise-first. - Fantastic.
So, run us through it. - Sure!
So, it's a health insurance company, so, as you can imagine, there are a lot
of near real-time transactions coming in, different data sources, we have roughly
20+ data sources, and about 40 terabytes of data that are coming in.
Or something is happening to them on a daily basis.
So, from these data sources, they streamline into this operations database.
And this is our operations environment, that is handling operational for transactional purposes.
- So, last time you were here, you talked about how you're using serverless with data
to do some processing. But, I see now you're using
some additional services, like CloudFormation. What's that for you?
So, we're using CloudFormation and SQS in hand, so everything kicks off
and is deployed using CloudFormation. So, our ingestion acquisition pipelines
they get triggered from CloudFormation. So, either it's trigger-based, event-based,
or time-based. And depending on that, CloudFormation will either kick off
a Redshift instance, or it will start Step Functions, or it will start Lambda,
or any other service that we have. - So, it's a great way to use the AWS
services, only when you need them... Reduces cost, which is fantastic.
But, I also see you're using Step Functions now. Tell us a bit about that.
So, last time we spoke, Step Functions was not available, now it is available.
And it fits in perfectly with our event, trigger-based architecture.
We can do things like rules, we can do things like events, or timing. You can even embed if-then-else logics in there,
for different usage. - So as the number of event sources has increased,
so have the rules. And it's a great way to get some extensibility
with your architecture, while preserving all that business function in place,
right? - Exactly.
So, the other thing I remember from the architecture...you had one place
for all of your data, using S3. That's great...tell us a bit about that.
So our environment has definitely become more complex, and we're more into the data management space.
With the help of our technology partners, we're able to create this analytics environment.
And each environment is an abstraction, and it has a different purpose.
So, you have an analysis area, a core strategic area, and an insights area.
And all of these, some way or somehow, are to add value and support
our operations environment, so that we can do some good in the world for our members.
So, you have, for example, Redshift, for this first environment.
You'll have Step Functions for the second one, for the core, and you have either Lambda
or Step Function, these can be interrelated... It's not fit.
So, S3 is awesome because, you know, it's a bottomless data store.
What was the reason for the segmentation? The segmentation...it's to ensure and insist
on data quality, data integrity. And there are different aspects of that.
So, we wanted to ensure data consistency. So, if any of these arrows here
or any of these pipelines fail, then we'll have SQS kind of jump in there
for exception handling, for messaging logging. The other reason for having
these different abstractions is to utilize IAM service, for more data protection.
So, more isolation...obviously, you're extending some of the event driven capabilities of S3
to kick off those different parts of the pipeline.
- That's right. - Fantastic.
So, I see that you've also started using machine learning since we last spoke to you.
Yes. Tell us about how that is complementing everything.
So, now, these different areas. You have your analysis area, where we want
to get data into the hands of our consumers, are end users...that's one consumption mechanism.
For our core strategic efforts, we're using Glue throughout.
So, you have Glue for ingestion, Glue for transformation,
and you also have Glue for the load. And we're using the catalogue service as well,
which has been exceptional. And then, for this insights area,
we're using machine learning or AI-related AWS services, such as SageMaker,
SageMaker Notebooks. And, essentially what we're doing over here
is we're scoring all that data that we're receiving from the left side.
So, you're reusing a lot of the data that you're collecting, learning more from it,
and it's adding to the scoring capability of understanding your customers better.
Exactly. So, what we're doing over here... if you want to think about the first layer
as kind of an R & D, and then from here, those creative insights, or creative thoughts,
start streamlining into some of our core business values, core business strategic initiatives.
And then from there, once it gets tried and trusted, then we can bring it into this insights environment, insights abstraction.
And, that's where our data science team, data engineering team can use some of these AWS services.
And then we can bring it back into this operational environment.
And from there, that streamlines through other third parties,
and we get data into the hands of our company.
And then we do good for our members. That's fantastic.
It's great to see this serverless data pipeline grow so big,
and using the event-driven architecture... but also extending to use these, you know,
purpose-built data processing services from AWS. Bilal, thank you so much for sharing your architecture
with us today. - Thanks for having us.
And thank you for joining us on, "This is My Architecture."