Hi, my name is Ryan, and welcome to another edition of
'This Is My Architecture.'
I'm joined by Sergey, Director of Engineering for Zulily.
Sergey, tell us a little bit about Zulily.
Hey, Ryan. Zulily is an e-commerce retailer.
We offer amazing deals on unique products for moms, kids, and babies,
and we do it in a fun and engaging way.
Excellent. So, tell us a little bit about the business problem
that you were trying to solve with this architecture.
Yeah, we were trying to create a suggestive search experience
for our customers that's inspired by what other people do in our apps.
So, let me walk you through this.
So, we start with the customer being on the mobile phone,
and as customers navigate the app,
they view pages, they click on things.
We record all of that in AWS Kinesis. This is our clickstream pipeline.
Now for search, specifically, we only care about search events.
And so, we put together a filter
on top of this clickstream dataset that's in real-time
through a very simple SQL statement,
filters out search keywords out of this massive data set of everything.
And then it chains it up and writes it to another Kinesis Stream.
And so, this is our way to basically do this filtering
with no servers involved, and, you know, at scale.
Excellent. Very cool. Okay, so you're doing some filtering with Kinesis.
Tell me a little bit more about this next phase of the diagram.
Sure. So, once we've got our search keywords identified,
we want to look at corresponding brands and product categories for those.
And so this is what Transformer implemented in AWS Lambda does.
For every search keyword, it makes a call to the Search API
and brings back related brands and product categories.
And then the second thing it does, it writes a document
to our data store which is implemented on top of DocumentDB.
Excellent. DocumentDB. Okay.
And tell us a little bit about some of the metrics around the documents.
How big are the documents?
What are your transaction throughput numbers, etcetera?
We have around 650 million documents sitting in this database,
rounding about less than 1000 TPS, typically, transactions per seconds.
We've got a seven-day TTL on those documents.
And, just probably another factoid, we are running three availability zones,
so there's a primary and two replicas.
Excellent. Okay, great.
And then, before we jump to the next part of that,
you know, I'm sure when you guys were evaluating the architecture,
you made some trade-offs and looked at some other technologies.
Can you talk through some of that?
Yeah, our team is actually well-versed in Mongrel,
and that was our first suggestion for this particular feature.
But then we realized, you know, DocumentDB is available,
it provides full compatibility, and really it came down to
what we wanted to focus on,
business logic or the database.
And so I remember, I was having a conversation
with one of the engineers, and I said, "dude, do you want to be a MongoDB expert,"
or do you want to be a search expert?"
And so he said, "well, of course I want to be a search expert."
And that kind of was the final straw in the argument,
and we went with DocumentDB.
Excellent. Very cool.
So, that was DocumentDB and how it's integrated with Lambda.
So, talk to us a little bit about this third phase of the architecture.
Yeah, the third phase of the architecture is--
So, this Lambda writes to DocumentDB.
We've got another Lambda that reads from DocumentDB.
This Lambda runs on schedule every x minutes.
It pulls the data from the DocumentDB...
And for every search keyword, for every brand
and for every product category, it does a look up against an inventory API.
This is super important for us because the nature of Zulily's business,
we have products in our apps and on the site
available for typically up to 72 hours, and then they rotate.
So, we must ensure that we have those products in stock
before we can sell them, before we can offer them
in our search suggestions.
And so, this Lambda does the check and then once the check is complete,
it writes data to Elasticache for Redis
and that in turn, it goes to our...
customers on their mobile devices through a search API.
Very cool. So we've got multiple Kinesis services that are running
and doing some of the filtering for some of the data
that the users are typing in.
We've got Lambda to help transform that and integrate that with
some of that data into DocumentDB.
We've got Lambda, which is cross-referencing
some of that data with the inventory that's on hand
and then Redis on the backend to integrate with the search API
and return the results to the user. So, very cool.
Sergey, thank you so much for sharing your architecture with us.
And thank you, everybody. This is Ryan from 'This Is My Architecture.'