[MUSIC PLAYING]
RON PANTOFARO: Hello, everyone.
Welcome to our Google Cloud databases fireside chat.
Today, we'll be hosting Jono MacDougall, founding engineer
from Ravelin, to learn more about how
they're using our databases, and specifically Cloud Bigtable.
From Google, we have our very own Andi Gutmans, VP
and general manager for Cloud databases.
And I'm Ron.
I'm from Andy's product team.
So just to kick us off, Jono, can you tell us
a little bit about Ravelin?
And Andy, you can take it from there.
JONO MACDOUGALL: Hi, Ron.
Thanks for having me.
Ravelin is a fraud detection company.
We do all sorts of different fraud,
such as credit card fraud, account takeover, account
takeover, supplier fraud, voucher abuse,
and a number of other different fraudulent areas.
And we do that for big e-commerce clients
as well as large enterprises.
So what our clients will do is send a request to us
when they have a transaction go through.
And we'll run it through our machine learning models
to figure out if this looks like something dodgy or not.
And then we'll respond to that client
and say, hey, OK, maybe you should check out this customer.
Maybe you should block them, or just
give it some more scrutiny.
ANDI GUTMANS: Jono, thanks for joining us today.
So what are some of your industry's technology
challenges?
JONO MACDOUGALL: So our industry,
I think the challenges are pretty
similar to other industries.
We worry about scale and we worry
about keeping our latencies low.
But because we are a fraud detection company, whenever
we get a client, we bring on the entire data and throughput
of that client.
So I think some of those needs are a bit exaggerated compared
to some other industries.
We don't get that sort of natural growth
we get sometimes in other industries.
So we really need to make sure that our throughput
and our scale can really cope with any time,
say, sales come in with a big client.
The other issue is, of course, we're
in all of our clients' checkout flow.
So they're sitting there waiting for us
to make a decision so their customers can buy things.
So we have to make sure that our latencies are low enough
that we don't interrupt that at all.
ANDI GUTMANS: And we all know that online retail has really
been booming over the last year.
So I can imagine the kind of scale
that you have to deal with.
So I know you're using Cloud Bigtable.
What kind of use cases are you using it for?
JONO MACDOUGALL: So we use Cloud Bigtable
to get data out every time we make a fraud decision.
So when a client comes in and one of their customers
does an order, we have to get the full history
of that customer.
We have to get as much data as we can about that customer
to generate features to try to detect if they're fraudulent
or not.
To do that, that's quite a bit of data
in quite a short timeline.
So Bigtable is quite good at that.
If we have sort of a key for the customer,
we can quickly get that specific data
and bring it into our feature extraction
to generate features for our models and our rules.
So that's kind of the one use we have for it.
The other use we have is for our dashboard.
We will present a customer profile in our dashboard
to our clients so that if we make a fraud decision,
we can have our clients look at that
and analyze if they think it really was fraud or not.
And they can use the same sort of data source
that we have for how we actually made the decision.
Occasionally, we also use it for some debug information.
We put some request logs and some trace information in there
as well, mostly because we have spare capacity in it.
It's just a really fast, nice place to put it.
ANDI GUTMANS: So what drove your decision
to use GCP and Bigtable?
JONO MACDOUGALL: So we had a bit of a journey to get there.
We were originally on PostgreSQL.
I think most startups will start with something like PostgreSQL
since it's easy and it's what you know.
But we quickly realized, scaling that's going to be a challenge.
And we didn't want to have to manage
these enormous PostgreSQL instances.
So we knew we wanted to go somewhere else
and we wanted to use some kind of key value store.
Because we weren't doing crazy joins or complex where
clauses or anything like that.
We were really just saying, hey, here's a customer ID.
Please tell me everything you know about it.
And that's where key value kind of really shines.
We used Cassandra at a previous company.
But managing that was a chore.
I know we had to hire a bunch of people who their entire job was
managing Cassandra.
So we wanted to go with something managed, is
what I'm trying to say here.
So we looked at Bigtable.
We looked at a few other solutions.
But with GCP, we were also really into BigQuery.
We were heavy users of BigQuery already.
And we really liked what it was doing.
We also wanted to go to Kubernetes.
This was about five years ago.
So Google was definitely one of the few
that were really helping on that side.
Now I think the other Cloud providers do it.
But I think Google is still kind of the top of that
stack as well.
So that's good for us, too.
There's a few other things we liked about Bigtable.
I mean, it has this versioning capability.
And we did do one weird thing where we did upserts,
and the versioning really helped us with that.
So I think some of the features were also important
there for making that decision.
ANDI GUTMANS: So we know migrations are super hard.
And they're especially hard in an always-on industry
like yours.
Can you tell us a bit about your migration journey?
JONO MACDOUGALL: Yeah, it's kind of terrifying, migrations.
When you want no downtime, it's really challenging.
So what we did was, we were migrating
not just from PostgreSQL to Bigtable, but also from AWS
to GCP.
So, to do that, we actually set it up
where we had a queue, which would sort of asynchronously
push every request we got over to GCP.
So we'd be running in AWS, just as we were previously.
But at the same time, we were sending requests over to GCP.
And we could then go back and look at those requests,
see if any were failing, see if the results were the same
as they were in AWS, and just look at response times
and things like that.
And we did that for about a month.
And so we could tune things just how we wanted,
as if it was live.
We then, one day, one scary evening, we
flipped a config flag.
And it was 100% over to GCP.
The other thing that happened was, at the exact same time,
it flipped the queue over to AWS.
So we were actually still sending traffic
into our legacy environment.
And the nice thing about that is, if anything went wrong,
we could fail back.
And we weren't missing data.
And we ran like that for about a month.
We never had to fail back.
But it was very comforting to know that if it all went bad,
we could go back.
So we were briefly multi-cloud redundant, which is nice.
ANDI GUTMANS: It's really impressive
how you pulled off the kind of seamless online migration
between clouds.
Kudos to you and the team.
What are some of the other things
you think our customers should know about Bigtable?
JONO MACDOUGALL: Yeah, so if you're using Bigtable,
one of the most important things to make use of
is the Key Visualizer.
I don't know if a lot of customers know about it.
And it used to be a bit hidden in the UI.
But now I know they've put it right to the front,
so it's quite easy to find.
And it's extremely useful.
I mean, if anything ever goes a bit wrong,
if you get some latency or some errors
coming back from Bigtable, it's always the first place we look.
Because not every time, but a lot of the time,
it'll be we have a hot key or a wide row.
And the Visualizer will just tell you.
And it will tell you the exact key range that key lives in
or that row is.
And you can go in and try to fix that at that level.
So it's really, really useful to know how your data is actually
hitting Bigtable and know if you're using any anti patterns
or if your clients have changed their traffic pattern
to exacerbate some issues.
So I think that's the primary thing,
I think, people should be aware of if they're not.
The other really cool feature is Garbage Collection.
It's really nice.
I mean, if you find a big row, it
makes it very easy to make sure that doesn't happen again
because you can put some limits on how big they can actually
get with the garbage collection policies.
And the other thing that we use quite a lot is the versioning.
Bigtable has a sort of 3D--
it's kind of a 3D array.
You have your rows, your columns, and then all
your cells, which are the different versions.
I think it's really common for people
to just use sort of the latest [? N ?] filter
and not really look at any of the previous versions.
But you can make use of that versioning to get history.
You can do clever things.
You can reverse the time stamp.
There's a few patterns in the Docs that are quite useful.
So you can get--
instead of the latest thing, you can actually get the oldest
thing if you need to.
So I think the versioning stuff, there's a lot of cool patterns
you can do with that.
ANDI GUTMANS: Awesome.
Thanks.
I really love the Key Visualizer, myself.
I think it's just super helpful to kind of see
what the access patterns are and how to see which keys are hot.
If you could ask us for one more feature request,
what would that be?
JONO MACDOUGALL: We have quite a few Bigtable nodes
now because we're at a fairly big scale.
And it starts to end up costing a bit of money.
But our traffic patterns are such
that we don't always run at our hottest CPU or what have you.
So it'd be really nice if you guys
had auto scaling for Bigtable.
During our evenings, I think it could save us
quite a lot of money.
And I do know there's some open source stuff out there.
But standing that up and making that work
is always a bit of a chore.
And if you could just make it push-button in the dashboard,
we'd use it for sure.
ANDI GUTMANS: Thanks for the feedback.
As you know, we can do zero downtime scaling.
There is an open source project that helps customers
with auto scaling.
But your feedback is very well taken.
And watch out for this space.
We will definitely try and prioritize auto scaling.
And hopefully you will see some news from us
sooner rather than later.
So do you have any workflows that
span databases and analytics?
You mentioned machine learning before.
How are you using kind of the overall data state
that we have and data services to achieve your goals?
JONO MACDOUGALL: Yeah, so Bigtable
works really well for our online fraud detection
to sort of get one customer record and their full history
and operate on it.
It's not as good at getting, say, every single customer
record to train on it, or to say, give me
every customer with this very complicated
where clause, who only ordered something on a Tuesday
that had whatever.
It can't do any of that, really.
So we also use BigQuery heavily.
We essentially put the same data in both Bigtable and BigQuery.
We send that data asynchronously to BigQuery
using streaming inserts.
And that allows our data scientists to come in.
And they can mess with the data all they want.
They can query it in every way you can imagine and build
models and investigate patterns and all that sort of stuff,
and not have to worry about limitations
of the query engine.
It also means it doesn't impact our Bigtable cluster,
our production cluster, when they do that.
It's a completely separate cluster.
So them doing a BigQuery has no impact on our response times,
for example.
Back when we were on PostgreSQL many years ago,
that wasn't the case.
That was a large portion of our outages.
So I think BigQuery is really good in that regard.
We also use Elasticsearch for powering our dashboard.
And that's just to do sort of a bit faster than BigQuery,
but a bit slower than Bigtable, style queries.
RON PANTOFARO: Can you take us--
the engineer in me kind of wants to know a little bit more
of the nitty gritty details with regards to Bigtable.
So how you start and what kind of the schema looks like,
what you have for cells?
JONO MACDOUGALL: Yeah, so we do use cluster replication.
But I guess I'll first talk about how
our data is structured.
We actually originally had everything spread across rows.
So we used sort of a hash of a customer ID as a prefix.
And then we could scan each record of history
from that customer.
So it'd be an order, an order, a transaction,
something like this.
We could scan all that out.
But we found eventually we got customers that were too big.
And that scanning wasn't quite as fast as we wanted.
So we actually switched over, and we put everything
into a single row.
So you basically look it up by the customer ID.
And then each column is a sort of different entity
within that customer.
So you have an order column, you'll
have a transaction column, a payment method column,
things like this.
And then each cell will be a different record,
a different order, a different payment method,
different transaction.
So you can look up the one row and get all the details
of that customer you need.
And that's really fast.
It also makes it very easy to use Garbage Collection to clean
up any really big customers.
Or in our cases, we occasionally have
clients who send us test customers, who
do an order like every minute.
And that very quickly becomes problematic
if you want to pull out enormous amounts of data like that
without any limits on your row size.
So being able to use Garbage Collection is really useful.
We also use replication for reliability.
How we do it is, a single request
needs to have that consistency.
So within a request, we always hit the same replica
of Bigtable.
And if we have a failure and we need to retry,
we retry the whole request.
So that might involve a number of calls
to Bigtable to get different data and also to write data.
So that allows us to sort of make use of the replica
and also make use of some of the consistency guarantees.
So it's a nice little trade-off you can make.
You can kind of choose where you want your consistency to live.
ANDI GUTMANS: Jono, thank you lot for joining us today.
It was great hearing about Ravelin's journey
in delivering market-leading online fraud
detection with Google Cloud and Cloud Bigtable.
Thanks for joining us.
JONO MACDOUGALL: Cheers.
Thanks, Andi.
Pleasure.
RON PANTOFARO: Jono, thank you so much for being here with us
today and telling us how Ravelin provides fraud detection
solutions to its customers using our products.
And Andi, thank you so much for hosting.
On behalf of everyone here in Google Cloud databases,
thank you for watching.
And see you on the next fireside chat.