Hey, folks. Daniel Myers, Developer Relations, here at Snowflake.
On today's episode of "Powered By," I'm talking with Lior Gavish, Co-founder of Monte Carlo.
Monte Carlo is a data observability company powered by Snowflake.
Lior, how are you today?
I'm great. Daniel, thank you. Thank you for having me here.
I'm excited to talk to you today because it really seems like
what you're doing at Monte Carlo is pretty awesome.
Can you tell me more about the founding story of it
and how it's grown over the last couple of years?
So Monte Carlo is the data observability company, as you mentioned.
We started out two years ago, and what we do is we help data teams
manage the reliability, the quality of their data product.
So we have a lot of data engineers, data scientists, data analysts out there,
and they're building products that they're putting in the hands of other people
They're giving them analytics dashboards, machine-learning models,
data sets that are used in production or sold to customers and partners.
And we're basically helping them make sure that those products
are at the quality and reliability level that they need to be.
We started this company just recognizing that
there's been a lot of work over the last couple of decades
helping software engineers to deliver reliable products.
Today, when you log into Snowflake or Google, it just works,
and it works with five nines and whatnot.
But people that build data products struggle with that.
We started Monte-Carlo two years ago,
and we spoke to over 100 data teams before we did that.
We realized that everyone is struggling with, "How do I make these products reliable?"
"How do I make sure the people consuming the data actually trust it, and that it's working at a good level?"
We built Monte-Carlo to both build the discipline,
like the equivalent of site reliability engineering for data teams, and also to build the tooling, right?
And Monte Carlo, the product, is really a platform that helps teams operationalize reliability,
to understand when data is broken, to then resolve it as quickly as possible.
And in many cases, we've actually been able to help customers prevent downtime
in their data products from happening in the first place.
And it's been a great ride so far.
We've been very lucky to have partner with some great customers in the industry.
We've doubled our customer base and our revenue every single quarter since we started.
And we work with some great companies, including
the New York Times and Vimeo and Affirm and a few Fortune 500s as well.
So it's been a great ride and we're very proud to work with all these wonderful companies.
We've also been fortunate to work with great partners.
On the investing side, we raised over $100 million,
from funds like Excel and GGV and Redpoint,
And those are the same backers that helped
Snowflake and Looker start their businesses, and we're very excited to partner with them.
That's awesome. That's really powerful to hear.
And it really seems like it's a good time because as more and more companies are considering
themselves to be data companies, it means it's even more important to make sure
that the data that you have one, you can get insights from it
and making sure that it is reliable, right? So that's really cool.
Can we see a demo of this platform today?
Sure. Absolutely.
Happy to walk you through how
our customers experience Monte-Carlo.
And as I mentioned, our goal here is to help people understand
when data is broken and then act on it as quickly as possible
and even prevent it with the overarching goal to eliminate as much downtime as we can, right?
Minimize that.
And so the way most of our customers experience Monte-Carlo
in their day-to-days is actually through Slack.
So the first thing that a Monte Carlo user experiences
is a Slack notification that might tell them that something is broken with their data.
And the magic here is that they can get those Slack notifications
with practically zero configuration or set up or threshold setting.
Monte-Carlo takes care of it, so it automatically connects to the data infrastructure.
And a lot of our customers who use Snowflake are very excited about that.
So they can connect their Snowflake data to Monte Carlo,
and we automatically map all of the tables that they have there,
all of the dependencies and all of the metrics that are required in order to understand data help.
And then we automatically build machine-learning models that can determine
whether data is healthy and whether things are working as expected.
When it doesn't, we'll send a Slack notification or an email
notification or a PagerDuty alert, and let them know that something is broken.
So one example of that that I can show today is right here.
And this is from our own Monte Carlo on Monte Carlo account.
So this is our own environment and, in this particular case, we've identified that there's an issue
at a particular table in our very own Snowflake account.
The table here is called monitors.
And what Monte Carlo identified here is essentially that rows have been deleted from the table, right?
And this is a table, as we'll see in a bit, this a table that usually only adds data,
and it adds rows as time passes.
And we've identified a deletion, and that's something that,
in a production environment, can indicate an issue, right?
Why is data being removed?
And so I might get this alert, and then I can go into Monte Carlo and,
let me flip over here to the dashboard.
I can go into Monte Carlo and really understand what's happening, right?
And the first step when you get an alert is to understand, hey, what happened?
And this is what you see on the graph right here.
This is essentially the volume of data in this table over time, and it's pretty easy to see that
it's constantly increasing. In this case on this particular day, on October 12,
we've seen a little drop, which might be an indication of an issue, right?
The next step for someone using Monte Carlo is maybe to ask themselves, does this even matter?
Should I act on it? What's the priority of this issue?
And we try to help with doing that by providing a lot of context on how this table gets used and data products, right?
Maybe this is just a table that nobody uses and I can move on with my day.
Or maybe it's driving the most important reports in the company,
and I should be working on it right now and letting the people using
those dashboards know that things are broken, right?
And Monte Carlo will help you do that in various ways.
So for example, if you can see this little star here, it actually automatically lets me know
that this is a key asset. And this is something that Monte Carlo does fully,
automatically based on usage statistics, based on lineage that we automatically derive.
We're able to tell what are some of the most important tables that people have in their environment and their account.
This is actually powered by Snowflake.
We actually bring a lot data about our customers' data environment into Snowflake
and run analytics that help us automatically detect those key assets using machine learning.
So this is one indication that might help me understand this is actually an important issue.
The other thing here that might draw my attention is,
we provide visibility into what are some of the downstream products that
data flows into from this table. And this includes
very indirect relationships. Like there might be dozens and dozens of processing steps before
data from this table ends in that product.
But really knowing that this table impacts, in this case, the Customer 360 report,
which is our single most important report at Monte Carlo.
It's how we know how our customers are doing,
makes me think this is something that I might want to take a deeper look into, right?
And Monte Carlo will help understand all the changes in my environment
that could have contributed to this issue, right?
Like this drop that I am seeing in data volume.
It might happen because someone changed the code,
the queries that are generating this table.
It might happen because the upstream data changed
in a way that I don't expect.
Monte Carlo really gives a lot of visibility into that.
Here we can see the lineage view, which is automatically generated and helps me understand
where data is coming from.
And this table, it also shows me how it's directly used downstream.
It gives information about the queries that are writing data into this table
so I can quickly see changes in those queries.
And you can actually tell there was a change right here.
I can even go into that and see what the query is,
and what it was before the change.
And I can see a lot of additional information that might be helpful
when troubleshooting this table, including freshness, when this table was updated,
the changes in volume, any schema changes that occurred
in the table that might contribute to the issue.
And a lot of other information, including analytics about the data itself,
whether I'm seeing knolls or whether I'm seeing duplicates,
whether I'm seeing cents instead of dollars, things like that.
So all of that goes into the platform and this is all powered by Snowflake.
So these are all things that we bring into Snowflake, and we use it
both for serving to drive the application that you're seeing in front of you.
We also ... our data science team uses it to actually build those models, right?
So our ability to detect issues fully, automatically in real time, is actually
driven by by the great work we've done on the Snowflake platform.
So that's really cool.
So, when you mentioned at the very beginning, all of this
really starts at that notification, right?
How is it that you are finding these things in near real time?
What we do is, we basically take all of those metrics that we've
talked about that help us understand data health.
So it's data freshness, data volume, schema, the distribution,
duplicates, things like that, and lineage.
We bring all of that into a single platform.
And then we build models that look at past patterns.
So in this example, what do volume changes look like for this particular table?
And then we build a model that essentially uses that past information
to predict what we expect to see today, right?
And if we see that reality, or we're seeing an environment deviates from those past trends,
we're able to automatically generate that alert and
notify the customer that there's something to be looked at.
So that's incredible.
So how is it that, as you were going about building this product--
which, by the way, over the last two years,
it's incredible to see where it is--
how how did you land on Snowflake being a core part of your tech stack?
It was a pretty easy decision for us.
When we first started out ... so we actually work with a lot of data teams,
and we help our customers monitor Snowflake as well as other
solutions out there and other pieces of the stack, including the
AI tools and data lakes and orchestrators, like Airflow and DVT.
And so we got really good,
you know, a bird's eye view of what the industry looks like and
what people are using out there and how it fits into different use cases.
And we really got excited about
Snowflake as a solution for ourselves.
Several reasons.
So we are on our application, the AWS, and we knew we wanted
a solution that would run there, close to where our operational environment is running.
We knew we wouldn't want to be moving
vast data sets across clouds,
and we knew that we wanted a solution that will allow us
both serve production, use cases, and development exploration
for our data scientists with the same data sets.
But in a way, where the data science work doesn't impact
our production workloads, and it requires minimum management and scaling on our end.
We are a start-up.
We've only been around for two years, and we knew we were going to grow.
And we wanted to have a partner that can do that with us,
and that can allow us that flexibility with minimal operational overhead.
So that's really cool.
So, as you're building, continuing to build this platform,
what Snowflake features would you say have been the most critical
to the building of your platform?
Absolutely.
We extensively use External Tables.
That allows us to, basically, very easily get data
from our streaming sources, from our operational environment
into Snowflake and build our analytics
and machine-learning models on top of that.
So that's been incredibly powerful for us.
We've also used Time Travel extensively.
It's incredibly powerful when we need to troubleshoot an issue,
especially in production use cases,
to really be able to walk back in time and see what things
looked like when the problem actually happened.
And then something I'm very excited about,
I can actually show you what it looks like.
We've built what we call Insights, which is really the first solution
out there to help data teams with operational information
about the reliability, performance, and even the impact and the
effectiveness of their work over the data platform.
And the way we do that is
we create a lot of very valuable
data sets for our customers.
So for example, we'll help them
understand what are the key assets in their environment.
We'll help them understand where they're covered
in terms of monitoring and testing. We'll help them understand
where they're incurring data debt, where there's things that can be deprecated.
And many other things that help really operate the data platform itself.
And it's the first time that people can do it with actual data.
And of course, we made it available in our UI.
As you see here, you can download those reports.
It's also available through API. But what we're really excited about,
and our customers are excited about, is really using
Snowflake Data Sharing to put that information back into our customers environment.
Some of our customers actually have access to those data sets
in their own Snowflake account,
and they're really able to join that information with other data sets
that they have with metadata that they have, and really customize
the analytics and the solutions to their own needs and to
their own way of operating their platform.
And that's been incredible for our customers.
I love hearing that because I know Data Sharing on Snowflake is
one of those features that a lot of people, I think,
underestimate the power that it provides,
and giving customers control over their own data in that way--
having them be able to join that data with other data that they are also
gathering from their own product sources or even
or even data that they're purchasing from the Data Marketplace, right?
They can do all of that through Data Sharing.
So that's that's really cool to hear.
I'm glad that's a a big element for you.
So, moving forward, what are your personal goals for the company
over the next six to 12 months?
We started Monte Carlo to really make our customers happy.
That's first and foremost.
And to make a change in our industry.
We believe there's an enormous opportunity to help
really democratize data, and help productize it, and help make
it more accessible for people.
And so my personal goal over the next 12 months is to continue to drive that
and to continue the level of growth that we've seen.
And in order to do that, we need to build out our team and grow our team.
And we also need to go out there and improve our product in many ways.
We want to stay ahead in this market.
We want to build the best observability solution out there,
and really give our customers the best observability experience
and the best feature set to match their platform.
That's awesome.
And so for our viewers today, if they want to learn more about Monte Carlo,
where should they go online?
We're very active on LinkedIn.
So please follow Monte Carlo Data on LinkedIn,
or visit us on MonteCarloData.com.
Awesome.
And for all of our viewers here today, if you're looking to build
your next application or business on Snowflake,
check out developers.snowflake.com.
Thanks.
Thank you all.