Hi,
It's Emma,
Today I'm gonna talk about a very interesting topic.
How to crack the product sense problems for data science interviews.
Lots of people have found this kind of question very hard to prepare
because they are either fresh out of school
or they have been working in a different industry that don't have any real experience of working with product.
So they don't know what these questions look like.
how to prepare for them and how to answer them during data science interview.
If you are in a similar situation,
this video is for you.
I promise you that you could learn so much from this video because
I'm going to share the exact the frameworks that I use to answer product sense questions during interviews
which helped me get job offers from Airbnb, Lyft and Twitter.
I've spent numerous hours reading through books and blog posts and thinking and talking to people to the develop my frameworks.
I'm also sure my frameworks with a few friends which helped them get their dream data science jobs.
So these frameworks are proven.
If you could re-watch in this video and internalize these frameworks,
it will save you a ton of time for interview preparation.
Let's get started.
So here's the overview of this lesson.
We'll start with understanding what are product sends problems.
Sometimes they are called business case or metric problems,
but they refer to the same thing.
Then I'm going to show you the three most commonly asked categories of questions
and provide you the exact the frameworks that I used to answer each group of questions.
The three groups are: diagnose a problem,
measure success and launch or not.
Finally,
I'll share some tips with you to ace the product sense interview.
OK,
Let's dive in.
First and foremost,
What are product sense problems? Well,
they are just the product related questions
that you may be asked during the data science interviews.
Then why they are asked?
As Peter Drucker once said if you cannot measure it,
you cannot improve it.
And the most critical skills of a data scientist
include defining the right metrics to measure the success of a product of feature and be able to diagnose and solve real product problems.
While some companies,
such as Facebook,
Twitter and Lyft care about whether a candidate is a familiar with their products.
But all companies will want to evaluate
the problem solving skills of a candidate by asking these questions.
Well,
it sounds like you need at least a some experience to be able to answer this kind of questions.
But it's actually just like a skill that can be developed that like any other skill out there. And the way it comes is through practice.
What I mean by that is even if you don't have any prior experience,
you could still gain the knowledge by reading,
listening,
thinking and summarizing.
And if you are here are watching this video,
you are already on the right track to gain your product sense.
So what companies are looking for,
or what constitutes a good answer is basically three things: structure,
comprehensiveness and feasibility.
Basically,
a good answer should have a systematic approach to the problem.
Any it covers,
all important aspects of that problem.
Also,
the solution should be practical enough so that it could be implemented realistically.
Typically, product sense questions
are asked in both the technical phone screen and the onsite interview
for a data scientist position. And at least one round of interview will focus on products as questions during an onsite interview.
The first category is to diagnose a problem.
Typically,
you are given a scenario that is one of the important metrics has shifted to the negative direction and you are asked to figure out the root cause of the issue.
Here are some simple questions.
Creation of Facebook user groups has gone down by twenty per cent.
What will you you do?
We have a dashboard tracking our metrics and the average ETA is up by three minutes. How would you investigate this problem.
How to investigate the 10% drop of the Friends application.
For this kind of problems,
I believe everyone has some ideas to analyze it,
but why someone can get the strong yes?
while others fail the interview.
The key here is to show that you have a systematic approach to this kind of problems.
There are so many aspects that you could talk about,
but you don't want to make the interviewer feel that you are throwing up random ideas and thoughts at him or her.
So here's my framework to answer these questions.
There are six steps in total,
but not every question needs all of them.
First,
you want to clarify the definition of the scenario and the metric.
For example,
the ETA question, you want to clarify how the start time and end time are defined.
Secondly,
is about the time aspect of the change. Did it happen suddenly or progressively?
Then you could analyze whether it was due to some internal factors,
such as the data source,
the data collection process or whether there's any bugs in the production code.
There are also some external factors to be considered like a seasonality.
The industry trend,
whether competitors
have done any marketing campaign recently,
or was there any special events or natural disaster?
Around the same time,
the metric has changed.
The third factor is about whether other products or features by the same company have the same change.
For example,
you could ask the interviewer: Have we made any change to offer a product line?
Have other related products experienced the same change?
Then you could further a segment by user demographics and behavioral features looking into regional, language,
and the platform。
For instance,
was a decline happening in an isolated region?
Did this change only involve iOs, Android or Web users.
To continue with a more in depth analysis,
you can even decompose the metric.
For example,
DAU daily active user equals to existing users
plus new users
plus resurrected users,
minus churned users. So you could examine which user group was the root cause of the change.
Finally,
it's important to summarize your approach to show the interviewer that you have a clear and structured way to analyze the problem.
To make the answer even better, you could add,
what do you think will be the most reasonable causes and how would you fix them.
The second category is how to measure success.
You are asked how to measure that success or health of a product or feature.
Here are some questions that fall into this category.
How do you measure the success of a product?
How do you measure the health of mentions?
Facebooks app for celebrities,
How can Facebook determine if it's worth it to keep using it.
Instagram is launching a new feature how do you tell it's doing well.
To answer this type of question,
I recommend to start with asking clarifying questions to make sure that you understand
the function and the goal of the product.
What it does?
How it is used?
Who is it for? This in super important because it's a huge red flag,
if you start answering without clarifying the question first.
Also,
imagine how awkward it will be if you spend five minutes.
answering the question and the interviewer tells you that your understanding of the product is wrong.
Once you have a clear,
understanding on it
you could define the metrics to measure it.
My recommendation is to provide no more than three metrics.
number of booking and conversion rate.
In addition,
you'd want to provide one guardrail metrics
which is a metric than should not degrade in pursuit of a new product or feature.
For example,
cancellation rates and bounce rate.
Good metrics should also fit the context.
One reasonable metric might not make much sense in a different context.
For example,
I once got this question how to measure the success of a new job recommendation algorithm during an interview,
and the goal of this new algorithm was to increase user satisfaction of the recommendation results.
So in that scenario,
using DAU as the success of metric is not suitable.
It makes more sense to use metrics that
fit the context are better.
For example,
you can use the clicks through rates of the recommendation results.
Or you can use the percentage of users who actually applied for the job after seeing those results as a success metric.
A guardrail metric in that case could be the average time it takes to return results.
Because a good recommendation
algorithm should not only return to the results,
but also returned results quickly.
The third type of question is what I call launch or not.
You are asking how to test a product idea or whether to launch a product or feature.
Here are some examples How would you set up an experiment to understand the feature change in Instagram stories?
How would you decide to launch or not if engagement within a specific cohort decrees if all the rest increased?
What would you change in Twitter app?
How would you test if the proposed change is effective or not?
If a PM says that they want to double the number of ads in news feeds,
how would you figure out if this is a good idea or not?
After seeing these questions,
you might think why this category is called launch or not。
While some questions just ask you to test if an idea is a good idea.
The reason is that you always need to make a recommendation
There you could talk through
How would you design an experiment to test it out
which could include how to split the user randomly into control and treatment groups.
And how long do you think we need to run the experiment?
Afterwards, we need to make some recommendations on whether we should launch a product or not
based on the experimental results.
You always want to link the results of the initial goal and the business impact.
For example,
what does 0.1% increase in conversion rate translate to revenue?
Is it worth it to launch the product,
given all the costs?
While the perfect scenario is that the increases of success metrics are practically significant and we don't see difference in the guard room metric,
but it does not happen often in reality.
So the interviewer might ask you,
What do you do If you see some complicating results,
such as increasing the DAU but also increase in bounce rates,
then you would want to translate that to the impact to the user and the business.
Also,
it will be helpful to consider the short-term and the long-term impact of the launch.
One reasonable suggestion could be even with the increase in bounce rate
the launch could potentially bring in more users to the platform.
And in the long term the benefits will outweigh the drawbacks.
This sounds pretty simple right,
but in reality we want to get to the numbers and do in-depth analysis to make a recommendation.
Last but not the least,
I like to share some tips with you on how to ace the interview.
Before you go to an interview or do some research on the company. As I mentioned earlier,
most companies don't care too much about whether a candidate is a familiar with their products or not.
But I do encourage you to do some research
to get familiar with is a products. Ask yourself how to improve the products and what metrics can be used to measure the success of those products.
Doing this kind of research leads to deeper and ultimately,
better conversations in interviews.
During the interviews.
there are three things you want to pay attention to. First always clarify the question.
Make sure you understand the high level goal before starting answering.
The second tip is to listen to the interviewer.
You don't want to follow the frameworks I provided rigidly or blindly.
During real interviews, the more important thing is to listen to the feedbacks and expand or shorter your answers accordingly.
Finally,
when you explain your thought process,
it's better to write up the bullet points so that the interviewer knows that you have a structured approach to the problem,
and that is very important to get a strong yes from the interviewer.
Those are the three frameworks and a few tips that I promised to share with you.
I hope you find them helpful. Product sense problems might not as hard as you think,
but it does requires time to be good at it.
That is for today.
If you've enjoyed the video,
click the subscribe button to be notified every week,
when I release is a brand new video on getting your dream data science a job.
Thank you so much for being here.
Drop me a comment If you have any questions,
I see you in the next set of materials.