- [Announcer] Our next speakers are Dr. Stephen Galsworthy
and Erni Durdevic of Quby.
They will describe how Quby uses MLflow
for their ML projects, including for model development,
model management and ModelOps.
Please welcome Stephen and Erni from Quby.
- [Stephen] Hi, thanks for joining us for our talk.
Today we gonna be talking about reducing energy waste
in homes, using a combination of IoT analytics
and Machine Learning.
I'm Stephen from Quby.
I'll be joined later on by my colleague Erni
and we gonna be concentrating on automating
our end to end workflows.
We have a particular focus on sharing some tips,
we've come across as well as explaining how we use
MLflow and in particular Model Registry.
So to start off, I wanna share a little bit about Quby.
So Quby we're a tech company that's based in Amsterdam.
We believe in a world where living without wasting
natural resources is easy.
We have been focusing on this since around about 2004
and we concentrate in the areas of energy insight,
and control for residential buildings.
The scale of the problem we're dealing with
is that households use a large amount of electricity and gas
across the EU and across the world in general.
Question we really ask ourselves here is how much
of this is just wasted.
So Quby, we're not here to tell people how much to consume,
but to make sure that all their use is intended
for the purpose that they really want
to use that energy for.
How do we do that?
So, we have a number of technology solutions in the sense
you can see how our smart thermostats and energy display,
as well as a number of our app solutions.
We are partnering up with some
of the largest utilities across Europe.
So we work with electricity and gas retailers,
as well as insurance companies and water utilities.
To give you a little bit of a flavor
of our technology stack.
We're collecting a large amount of IoT and customer data.
So this can be electricity meter signals coming in at
every ten second resolution, water data,
data from the heating system within the home,
as well as user interaction data, profile information data
that the user is willing to share with us.
We store all of that naturally in Cloud we work
with AWS as our provider.
Then we offer a number of machine learning algorithms
on top of that data.
That's where we working with Databricks in particular
using ML flow.
On top of that, we offer a number of personalized services.
This is in the areas of advice and insights,
control of the smart thermostats and home monitoring
services in general.
To dive a little bit more into details of that.
I'm gonna hand you over to my colleague Erni
and he will take you through.
- [Erni] Thank you, Stephen.
Yes, let's dive into our unified data analytics setup.
So it all starts with the IoT data stream that comes in.
And this is composed of more than a terabyte
of data per day.
So we start this into Delta Lake.
Delta Lake on top of AWS S3.
And over the time we have other data sources.
So SQL data and Click data streams that comes also in
our Delta Lake.
So on top of this more than three petabytes of data,
we run our batch processing.
So the batch processing takes a part of the data
that is in the Delta Lake,
extracts it around some transformation and stores it back.
And this process is iterated several times
until we refine the data with the point
that can be served to the user.
This batch processing pipeline is composed of
more than 60 ETLs
So some of these ETLs are simple data filtering
or transformations.
Some of them contain some machine learning models
that apply on top of the whole data set
and some contain a lot of machine learning models
that are trained for each individual customer.
So for instance, we run daily more than a million
unsupervised models that are retrained on all customers
for the past 90 days.
So in this way, we iterate through the process
until we get data that we can serve to the customer
through our services API.
So the services API needs to provide data first
with a queryable data source.
So this way services API are responsible
to store data in their own SQL or no SQL database,
in order to have a lower latency.
So let's dig a little bit deeper into
how batch processing pipeline looks like.
So this is an example you can see on the left.
We have the source data,
which is the IoT data that we collect.
We run some preprocessing and we start back again
to Delta Lake.
And from there on we extract the different signals
like electricity and water from this collection of data
and we store it back into Delta Lake.
And then we may then run some algorithms
by combining multiple sources.
And we start back and the cycle continues until
the data is ready.
So every transformation in this chat has the same structure.
So it has some data that comes in,
which can be one of more data sets
and have a data set that comes out.
So this is perfect for unit testing.
Let me show you an example.
So here I defined a data schema with two simple columns
an empty and a value,
and then defined the transformation on top of a data set
that has the schema.
So you can see that transform method accepts an equal
data set and returns a data set with the same schema
in these examples, but it can also be a different schema.
So a unit test would look like this.
With a find the input with the sequence
of data that we need to test our functionality.
And we convert these into a data set.
And then we create the expectations or the expected sequence
of what we can expect as a result,
but we don't need to put that into a data set
because we don't need Spark
for checking that the output is correct.
And then we run the transformation by passing the input
and we trigger the collect statement.
So the collect will execute all the transformation
and return the finds and return the collection back.
And then we compare this result collection
with the expected collection by comparing its length
and the distinct values that are in both data sets.
So these because Spark doesn't guarantee
the ordering of data set
and in order to prevent having missing mesh,
we just cast it to a set and we compare
and it will compare element by element.
So since Spark tests are generally pretty slow,
we here are some tips to make them faster.
So we reduced for instance,
Spark SQL shuffle partitions from a default of 200 to four,
which is more than enough in a unit test,
but of course, in terms of basic production.
And also we enable the parallel execution of the unit test
this speeded up in our case for more than two times,
take any test.
So as you have seen, we have some transformations data
in and out and they work on batch.
But most of them can also work on streaming.
So it really depends on how the transformation is defined,
but most of the time it can just be up with one to one
with the same code and get on a stream.
Most of the transformation work like Select,
Filter, WithColumn, GroupBy, the Join,
user-defined functions and also the Time-based windows
that was in the group by statements.
They all work in batch and streaming,
but streaming doesn't support unfortunately,
the non-time based windows like the functions lead,
lag, first or last in your data sets.
So if you have this type of functions
then it won't work on the streaming,
they work only on batch,
Usually there is also ways to rewrite,
this algorithm is into GroupBy segment
using time-based windows, if you use a time serious data.
But this creates a little bit different paradigm
of handling the data.
So it depends on the case.
So going back to our unified data analytics setup,
we have seen how we run the batch processes
but we also run R&D Notebooks.
So data scientists created the same data
that is used by the staging or production jobs
and they can execute the Data Science Notebooks
and train data, create future data sets.
And when they are satisfied, I'll not ask for that,
when they train their models, they log the performance
of the models and the training instance itself
into MLflow.
So MLflow at the moment contains more than 500 models
we have logged over time.
And these and when we are satisfied enough with the model,
we store it into the Model Registry.
So MLflow Model Registry has us providing models
to the batch processing that is happening in production.
And at the moment we have more than 15 deployments
tracked for instance.
And then on top of Delta Lake,
we also run real time services and live dashboards
to complete our stack of the batch streaming,
R&D, on top of the data, along with the model deployments.
So there are three crucial aspects that were simplified
when we apply MLflow and the MLflow Model Registry
to a use case.
So first of all,
it became easier to choose the right model
between different trainings.
Second, it is easier to find what is the model
that is running the production or the latest one
that has been developed.
And third, it's easier to manage deployment cycle
of the models.
Let's see some examples.
So, we believe that the source code of the model
is not just the code,
but it is the combination of code, configuration and data.
So how did we bundle them together before.
We used Notebooks.
And we used a different Notebook for a different combination
of input data code and so on.
So choosing the right model was pretty hard,
because you will have to go into these Notebooks
and check the matrix and compared them or remember,
which one was the best one.
So with MLflow, this is a bit easier now,
because we can see a list of all the training differences,
with relative parameters that were used in the training,
the metrics that we measured after performance
of these metrics and some custom tags
that we cannot ourselves.
And then it's also easier to find the latest model,
because previously we used to go to the data scientists
that trained and then more than nothing
where is the Notebook they used to training.
But now we have a unified interface,
with the MLflow Model Registry,
that we can use to see exactly what is the version
of the model that is deployed on which stage
and also click through and see the code and the data
it was used its training time.
And third we changed our deployment of the models.
So in this case, how we used to do it,
we used to create a deployment bundle
and this bundle would contain the artifacts
of the code from our repository,
would contain the job definitions,
scheduling, configuration.
So everything that was needed to run this job.
And it would also contain as part of the configuration,
the model path which was stored on S3.
So this whole deployments bundle
would contain all the information needed
for a specific cluster to run this job.
So if this bundle was deployed on production cluster,
then the production cluster would know
how to get the model from history, given the configuration.
But with MLflow, we don't store this information
into the deployment bundle anymore,
but we let MLflow Model Registry decide,
which is the right model for each environment.
So the production costs that will just request
the production model, the staging cost will request,
the staging model and MLflow would be responsible
of selecting the correct one.
And with this I would like to give the work back
to Stephen for closing notes.
- [Stephen] Yeah, so as Erni has just mentioned,
I think there's three areas where really MLflow
has been giving a huge impact to Quby.
What is that ultimately resulted in?
The first thing is really data science is getting more
impact across the entire organization.
So MLflow in combination with the other tooling
that we use is allowing data scientists, engineers
and analysts working across different product teams
to collaborate and reuse models.
The way we work at Quby is we work in a system,
what we call a value stream,
where we're actually having a product team
that is formed of multiple different disciplines.
So covering everything from development, data science
and engineering, all the way through
to product marketing and user experience.
Being able to spread our data capabilities across
the organization has really allowed us to develop many more
products based on data
and to get those in the hands of our end users.
Another key thing is MLflow facilitates a more uniform
way of working.
So as Erni showed previously,
there was many different ways of generating models
and managing model performance.
MLflow offers more uniform way of logging
the metrics associated with models
and tracking that model performance,
as well as being able to make a direct link back
to the data that a model has been trained and tested on.
Overall, this means there's less time spent
on infrastructure.
And from my perspective, as the Chief Data Officer of Quby,
that means the data team are adding more value,
that they're not having to spend a lot of that time,
just managing the infrastructure
they can really be working within those product teams
and contributing towards development.
What we also see is development cycles are quicker.
Whereas previously, work that had been done in the past,
particularly on ML model development was largely hidden.
We can see it's a lot quicker to pick up things
that proof of concept, we developed a number
of months ago and really rapidly bring that up
and be able to use that to increase the development cycle.
Overall MLflow, plus a lot of the whole MLOps way
of working is helping to power Quby's transformation
into an AI first company.
So you'll see this happy bunch of people are very much
passionate about what they do.
But ultimately the impact we're making is on the end result,
which is saving energy in homes.
So to give you a little bit of a flavor of that,
our Waste Checker service,
which detects inefficient appliances.
We've detected over 87 million inefficient appliance cycles
from a washing machine, dishwasher and dryer,
as well as a huge amount of waste identified and targeted.
Another product of ours called Thermostat Program Advice
just in the last eight weeks has identified a huge amount
of gas being in used in homes,
when there was no activity detected.
These users are automatically alerted and are able to modify
their heating program to better match
the habits of behavior.
So overall, we're very happy to be step-by-step
enabling this transition to a sustainable energy system.
So we hope you've enjoyed a little bit
of the run through we've given you today,
Erni and I will also be giving a talk
at the Spark and AI summit,
that will be taking place towards the end of June this year,
where we'll be covering in more detail,
some of these topics as well as touching upon
how we've been using Delta Lake within our setup.
So we hope if you've enjoyed this talk,
you will join us again then and learn more.
Thank you very much.