In 2006, Netflix announced the Netflix Prize, a $1 million-worth data science tournament.
The winning team had to provide a machine learning model that would increase their movie-recommendation
system accuracy by 10 percent. Three years later, the prize was paid... But Netflix didn’t
use the algorithm. They considered that the gains were too low compared to the effort.
Hello, I’m Alex Konduforov, Data Science Competence Leader at AltexSoft.
And today we’re going to talk about estimating return on investment in data science and machine learning projects.
Estimating ROI in data science is hard.
These types of projects face a lot higher uncertainty than traditional software engineering.
And if you’ve ever worked with engineering projects, you may yell at me -- there is enough uncertainty.
So many of them fail to deliver value or go over budget.
And that’s even worse with data science.
So, what’s the difference?
You see, data science projects fall into the R&D category -- research and development.
We’re usually fine with estimating the D, when we have an exact specification and understanding what the system should do.
But this R is a real pain! Research means that we have a lot to explore
and experiment before we arrive at a good algorithm.
At this point, we can’t say how much time it will take.
We may also realize that deploying the algorithm is too expensive and the gains don’t justify it, just like the case with Netflix.
And sometimes -- the worst case scenario --
we may realize that the implementation isn’t feasible at all. We just can’t do it.
Because of that, estimating ROI in most ML projects should happen after a data scientist does some research.
Have a look at the basic formula for ROI. It’s simple.
You subtract the cost of investment from the gains and divide it by the cost of investment.
There you go. But you need to calculate what goes into these two variables.
And this is the hardest part. Let’s start with understanding gains.
ML has a handful of specific cases where it shines when put against rule-based systems.
What kinds of problems is machine learning good at? Let’s mention the two general cases.
First, is optimizing business processes. You can do it by automation, augmenting humans
with better tools, or making some sort of a decision support system.
For example, your insurance agents have to manually process claims.
If you can decrease the time for manual processing by 50 percent, you can do twice as many claims during the same time.
That’s your gains that can be converted into a financial value.
The second one is building a customer-facing application.
For example, you make a recommender system that suggests products or personalizes news feed.
This way you increase sales, and engagement.
And eventually you have higher revenue. That’s you gain with customer-facing applications.
You still can solve some of those with traditional programming.
And yet, you have a higher chance to get better results with ML.
But the approach of projecting gains in data science is also different.
In traditional software engineering, we know what system can do and what it cannot:
We explicitly define requirements, acceptance criteria, and then code these rules.
But machine learning algorithms are solving more complecated problems and
they are for making some mistakes.
If we can understand how many mistakes the model is going to make, we can estimate the lowest threshold of mistakes.
Let me give you a quick example.
The main metric for ML classification models is accuracy or the percentage of cases when the model makes the right decisions.
Imagine that your insurance model must correctly verify 80 percent of the cases without your agent’s involvement.
This means you’re looking for 80 percent of model accuracy.
This is the most threshold that justifies building the model.
If you go higher, your gains will be higher as well.
80 percent maybe enough. If you make suggestions for next purchase.
But sometimes your lowest threshold must be 99 percent or ever higher,because your model
because your model helps doctors and makes critical decisions.
It all depends on your case.
So, to make the initial gains estimate, you should set up the lowest threshold of your model performance.
If you go lower, the algorithm isn’t worth doing, if we can go higher -- great.
You can get larger gains.
Great, But what about cost? How do you approach cost estimations?
As I said, you can’t calculate cost right away. You need some research which aims to exploring two key things about an ML system.
Data quality. And feasibility:
Can we build the system at all, given the expected model accuracy?
If we have good data and realize that the project is feasible, we can estimate ROI. So, what should we do?
The first thing is data exploration.
As you all know, machine learning projects heavily rely on data and its quality.
At this point, we want to make sure that there’s enough data, it has all relevant variables, and it’s labeled.
So,let's discuss this questions one by one.
Do we have enough data?
For instance, if we want to recommend new songs in our music app to customers, we must have enough user behavior data collected.
And if we don't, then our model accuracy will be low.
Does the data have variables with enough predictive power?
Let’s stick to the music app example.
We may know which songs user listens to, when, and how often.
But the real predictor can be whether the user listened to the full song or only a few seconds of it because they didn’t like it.
And we may not have this data. Is our data labeled?
Labeling means defining the value that we want to predict in our data.
If you have data on customer churn, you know subscribers who churned and who are still with you.
But this may not be the case for image recognition projects.
For instance, if you want your algorithm to analyze x-ray images of lungs, you may need healthcare professionals to label tumors, pneumonia, or other conditions in your dataset.
We also have to estimate how much time it will take to prepare and clean data:
Remove incorrect records, standardize the ways data is recorded, fix missing data, avoid bias, and so on.
Okay, let’s assume we have enough for data. And we can move forward with modeling.
Of visibility study we can do just basic modeling and experiments.
Basic modeling means using simple approaches to understand how well they performe on existing data.
Most likely such simple models won’t achieve the required level of accuracy.
But they help assume whether more complex ones will.
For instance, if a basic model shows 70 percent of accuracy, we may assume that using more sophisticated algorithms
and model ensembles will deliver up to 90 percent.
If basic modeling confirms that the system is feasible, we can procceed with team, and budget estimation.
Finally, you need to calculate the costs of deploying the model into production.
How much data engineering and infrastructural development will you need to make the model run?
All these research findings give us understanding of the gains,understanding of the costs
and finnaly allow us to estimate ROI.
Great. To sum up.
The key challenge in machine learning projects is their high uncertainty.
Because of that ROI estimates can be realistic after some initial research.
Plus we should consider the risk that the requested accuracy won’t be feasible at all.
Okay. But let me finish with some final recommendations.
Things that you should consider when estimating ROI for machine learning projects.
First, define the KPIs to measure your success.
For instance, if you want to automate manual operations,
think of how you can measure the baseline and compare your new results with it.
Will you use time, the number of completed tasks, the number of employees needed to complete the work?
Second. Test your algorithm in production.
The real results maybe different from what you expect.
And you initial arrive estimation maybe too optimistic.
In this case consider getting the real feedback and improving the model.
Finally, Improve in iterations.
Start small and try reaching your lowest accuracy threshold in first stages.
And then consider the cost for futher improvments and wheter you need them.
Don’t strive for the highest possible accuracy, because you risk spending too much.
To understand how the serves, have a look at this graph.
It shows that the costs increase as you gain more and more accuracy.
But at some point, the accuracy gains start diminishing. They approach some sort of a ceiling,
never reaching 100 percent of accuracy, no matter how much money you pour into a project.
When you estimate ROI, you always have to deal with this tradeoff.
You should understand the lowest threshold of accuracy and make incremental improvements before you reach the plateau.
Once you’re there, further improvements don’t make much sense, since every additional percent
will cost increasingly more. Even more so, in many non-critical cases
you may stop even earlier and have a solution which generates enough profits without spending too much.
That’s it for today. Thanks for watching. I hope you liked the video.
If you did, give it a thumbs up, check my other videos on AI and data science,
and see you in the next one.