Welcome to This is My Architecture.
I'm Tom from AWS, and today I'm joined by Mike from Homesite Insurance.
>> Hi Tom. >> Welcome Mike.
>> Thanks. >> Thanks for joining us.
I know you have a company policy where you can't endorse any specific vendor.
So, we thank you for joining us today.
Yeah, that's correct, Tom, thank you.
So tell us a little bit about Homesite Insurance.
Homesite was founded in '97 around the business model of
providing the best online experience for purchasing home insurance.
And, a large part of that model is bringing in leads from
major auto insurance and other insurance providers...
... for whom, we then, bundle auto and home insurance and other products.
And, you white label that service for all of these different companies?
That's correct. We call them our partners,
and we provide a seamless experience from their UI to ours.
Interesting.
So we're going to talk about your architecture today.
We're does the data start?
Yeah, OK, so, we have these UXs,
these user experiences that we provide to bring in these leads,
and to, you know, move towards a quote.
So, first of all, there are many experiences,
that have to be designed around
the specific requirements of the different partners.
Then, on top of that, we also have many layers of the solution
that are generating data at different granularities with different schemas.
And what we wanted to do, is to provide one cohesive vision of this data
that would enable us to improve our service,
and also to understand better what works and what doesn't.
Fantastic and so, you're really trying to build a comprehensive model of a user,
that could be coming from any of those white-labeled sites, at different times.
Yeah, that's exactly right, Tom.
Cool. All right, let's jump into it.
Um... where do we start?
OK, so we start actually, to the left of this whole thing,
with serverless data pipelines based on Glue...
... that are coming in, and when they complete processing,
what's happening is, they're writing events to our SQS,
which we use for orchestration between our stages.
So the nice thing is, that SQS will then natively-- can trigger Lambda.
So, we're triggering, what we call our, Job-Initiator Lambda.
And, what's happening then, is, the first thing it does,
is it reads a pipeline definition from our DynamoDB.
Now... we do two things, mainly, with Dynamo in this architecture.
One is, we store our pipeline configurations here,
so that these stages can be reusable and easily deployed
by just writing new records to Dynamo.
>> Nice. >> And the nice thing is,
because it's a NoSQL database, we can have those defined as JSON
which means that our schema is easily adaptable.
We also use Dynamo to write exit status of jobs to help
with the orchestration and reporting on the whole pipelines and their health.
Nice, so what does the pipeline actually kick off?
OK, so... when this Lambda has read, not only the pipeline definition,
but also the exit status of all of these other pipelines,
that happened over here, to the left of the board...
... it will kick off a step function.
A step function is at the core, of then, the coordination of the detailed steps,
that execute to load the data to Neptune and do the other operations,
that we're talking about here.
And you need to use step functions because you need to save state?
Or what's the driving--
Well, because, as you'll see when we get to these other steps,
there are a lot of detailed actions, some of which may take a period of time,
that would exceed a Lambda execution window,
and that really need tight coordination.
So, what we did here, is, in order to achieve the mission that we set out to,
when taking this web of data,
which really is kind of a web of data,
where you have a customer, they may have multiple quotes
that they initiated on different channels.
They may hit a call center.
They may have, actually, touched multiple partners, UXs.
And, also all of those layers that I talked about,
of logging, Google Analytics, all of that stuff.
So, what we determined is, we really couldn't manage that,
in a traditional relational database.
So, a property graph, in a graph database,
was the answer to the business problem.
OK... cool... ... and so, what happens next?
Well, so, we load vertices and edges to Neptune.
But, before we do, we need that cluster to be powerful.
We need it to have all the resources,
that it needs to manage a fairly large volume of data going in.
>> So-- >> And, I'm guessing--
Because you don't know at any one time,
how much data is going to be coming through the pipeline.
Well, yeah, and we'll get to more sophisticated orchestration.
For now, based on our understanding of the data coming in,
we're using a boto3 call, from this step function to actually,
scale up our instance from an R5.xlarge to an R5.4xlarge.
And, the reason that we do that,
is because, while Neptune solves this business problem, very neatly,
we don't want to keep that huge instance running all the time.
And, the cost differential is over 50%
if we scale it up only when we're doing UTL.
Right, what happens once, after you scale it up?
OK, so we scale it up,
and now we've got all the horsepower we need,
we load our vertices and edges...
... but now, we also need to manage
broad-based data access.
OK? So this is an excellent tool for the data scientists to go in now,
using Tom Sawyer or other graphical UIs.
Explore the data, and then, they create Gremlin queries...
... which allow us to traverse this model and export data to S3.
So, from here... ... Gremlin... is actually...
... outputting data to S3, based on these queries,
the data scientists have developed.
We catalog that in Glue, and expose it through Athena.
And now we have the broad-based data access that we need, in a really cost-efficient way.
That's a really cool solution for getting
that visibility to a complex scenario of a user.
So you're using the power of the graph database,
and then exporting that visibility to a very cost-effective solution in S3.
And then you're doing the queries in a serverless way using Athena.
>> That's right. >> Very smart. I love it.
Thanks!
Thanks for sharing your architecture with us today.
I enjoyed it.
And, thanks for watching.
This is My Architecture.