Let’s describe what a data scientist does in their work by using a memorable example.
Audimax is a company that operates an audiobook platform. Their business model provides a free
trial for non-paying users with access to up to 30 minutes of daily content. Ken (a digital marketing
manager) approaches Greta (one of Audimax’s data scientists) with a request, saying:
“We need to build a machine learning model that helps us convert more free users to paid users.”
Greta replies by saying that this is not a clearly defined problem,
and there are many possible ways to improve the free-to-paid conversion rate of a company.
She talks to several individuals in the company to formulate an opinion. In meetings with the
product manager and marketing and finance teams, Greta’s knowledge about the product and business
allows her to ask the right questions and reason with these teams . Then, Greta and
Ken reformulated the problem: Ken’s benchmark data showed that the company underperforms its peers.
Audimax’s free-to-paid conversion rate needs to grow from 5% to the industry standard of 20%.
Data science work takes a lot of time, and experienced data scientists initially consider
new requests with healthy skepticism. Greta explained to Ken that it isn’t a great idea
to jump straight into building a sophisticated machine learning model without reasoning whether
the model needs to be made. It’s always preferable to consider more straightforward solutions
first . If the result of a simple solution is almost as good as a sophisticated one, the data
scientist would more than likely use the simple approach before utilizing the more complex one.
The most straightforward tactic for Greta was to test a time-sensitive price promotion
since competitors were already doing so. Her hypothesis reasoned that a time-sensitive
price promotion—selling the product at 50% off to free users in the first five days
after registering—will help Audimax grow its free-to-paid conversion ratio from 5% to 20%,
which will double the firm’s monthly revenue. The data scientist instructed the development team
on implementing the A/B test. An A/B test is an experimentation tool consisting of splitting the
firm’s new free customers into two cohorts: one that will see the promotion and one that will not.
An A/B test aims to compare the results of the two groups and give evidence on which version
is preferable. While working on the A/B test, Greta had to consider such issues as sample size,
test duration, and power of the test . Results clearly showed that a time-sensitive
promotion produced a statistically significant effect. The cohort’s revenue with promotion was
50% higher , and the free-to-paid conversion rate of the group saw the price-sensitive promotion
jump to 10% . In comparison, the cohort that didn’t see the promotion kept the same conversion
rate of 5%. The hypothesis that a time-sensitive promotion would not have an effect was rejected.
Greta and Ken agreed that this was a change in the right direction,
but they needed to work on improving the conversion rate to catch up with their peers.
Greta suggested that she needed time to conduct an exploratory data analysis (EDA)
to understand better customer behavior and what could unlock additional value. It took
Greta several weeks to review the database, understand variables, clean various tables,
and analyze the relationship between variables through correlations and data visualization. Her
work led to an interesting observation. Free-plan users who engaged and listened to the product for
at least 60 minutes in their first three days were four times more likely to become paid customers.
Greta shared this finding with Ken and Patrick (Audimax’s product manager).
They agreed to run a test to unlock unlimited Premium content and exclusive features to free
users for 24 hours after registering. They hypothesized that this would drive up user
engagement, leading them to higher conversion. The results from the A/B test were encouraging.
Changes were applied to the entire population of free users.
After a couple of months, Ken told Greta that the firm’s free-to-paid conversion ratio had
grown to 15%. They were getting very close to the industry standard of 20%, but not there yet.
Greta suggested that simpler solutions had gotten them this far. As there were no other
apparent ‘quick fixes,’ perhaps it was time to work on a more sophisticated solution to
improve performance. One common topic that kept resurfacing in discussions with the product team
was that some users found the product’s base price expensive, while others didn’t . There
wasn’t an easy way to differentiate these users. This led Greta to think that perhaps they would
improve conversion rates and grow revenue even further by using an ML algorithm,
which predicts the discount amount clients need to convert. She began gathering the necessary data,
dealing with abnormalities and missing values , normalizing the data, the date ranges to be used,
figuring out the amount of data required, and all pre-processing work necessary to
start working on the ML model. Once that was done, Greta spent time on feature engineering.
Then she tested different ML algorithms to decide which would be optimal for her dataset.
For a month, Greta trained different types of ML algorithms and compared their levels of accuracy,
precision, and recall (performance metrics that indicate how well models operate). When Greta
decided on the model that would be used, she spent time finetuning its hyperparameters and improving
performance. The validation of the model showed some promising results. Greta was excited to put
the model in production. Because Audimax is a small firm and doesn’t have an ML ops engineer
on the team, Greta had to work with developers to productionize the model. She created a REST API to
spit out predictions quickly and live in the app. In this way, user experience wouldn’t be affected.
Then, finally, Greta set up an A/B test to ensure model results held when in a live environment.
After a couple of months, financial figures showed significant improvement,
primarily driven by Greta’s work optimizing Audimax’s free-to-paid
conversion. And then everyone lived happily ever after. 😊
We hope this example (among many) of what a data scientist does motivates you to learn
more about this exciting profession and bring about changes to help your company thrive.Thank
you for considering our video. Please remember to check out our 365 Data Science program if
you’re interested in a career in data science. (We’ve provided a link in the description.)