hey it's Emma welcome back to my channel since i published a video on cracking metric interviews a
few weeks ago lots of people told me that it would be very helpful to go through some real interview
questions and answers so today i will be talking about a real metric interview question from Lyft
i will first provide a few steps that could be followed to answer that question and then
i act as both the interviewer and interviewee to show you what it looks like in a real interview
i hope it can be helpful for your interview preparation let's dive in the question is we
have a dashboard checking our metrics and the average ETA is up by three minutes how would
you investigate this problem this question is a typical diagnosis of a problem type of question
and the target metric is average eta most metric questions are open-ended and there are many ways
you could approach them as a general reference i have covered a few steps you could follow to
answer this type of question in a previous video you could get the link in the description below
the goal of this framework is to help you get a sense of how this kind of question
could be possibly answered but if you could come up with anything else that's more related to the
question itself it's better to talk through that in the interview rather than following the steps
rigidly as that being said let's briefly recap the steps that i have provided
again if you are interested in learning all the details feel free to check the link below
specifically there are six steps the first step is to clarify the scenario and the metric to make
sure you fully understand the problem the next few steps investigates the issues from
different perspectives step 2 looks at the time factor and investigates whether the change has
happened suddenly or progressively then you could dive into each possibility we could also check if
other products or features by the same company have experienced the same change which may lead to
a much bigger problem and we will need to look at the metric of the entire platform
also we could segment the metric by user demographic and the behavioral features
to isolate the issue depending on the definition of the metric we could also decompose it finally
we want to summarize the overall approach to show the interviewer that we have a systematic
approach to solve this problem note that the order from step two to five does not matter that much
so you could definitely change it also you don't need to go through all of them you may
only investigate two or three of them depending on the feedback from the interviewer and the time of
an interview but the key here is to be systematic it's important to convince the interviewer that
you are organized and not throwing random ideas at him or her now you have a good understanding
on how to approach this kind of question let's see how we can answer it in a real interview
i'm gonna act as the both interviewer and interviewee to show you one possible way to
answer this question as well as the interaction between the interviewer and the interviewee
i'm going to be your interviewer today and let's start with the metric question
we have a dashboard tracking our matrix and the average ETA is up by three minutes
how would you investigate this problem okay so the average ETA has increased
how do you define ETA is it defined as the time it takes for the driver to pick up the rider
or the time it takes for the rider to arrive at the destination after he or she is picked
up it's the time it takes for the driver to pick up the rider after a writing request is sent i see
so how is average ETA defined is it measured by hour or by day it's aggregated by day
thank you for the clarification to start diagnosing the cause of this problem
i would begin by investigating a few factors to understand the context in which the increase of
average ETA was detected the first thing i want to check is if there's any extreme value that has
impacted the metric because average value is affected by outliers we can set the threshold
say 99.99 of all the ETA's and if there are extreme values that are much higher
than that threshold then we can further investigate what happened to those ETA's
such as if the data was collected correctly or if the large ETA's were due to some technical issues
but if there's no outliers then the cause is still there and we can dig deeper into the data does it
make sense yes it does so if there's no outliers how would you proceed with the investigation
sure if no outlier can be identified i will then focus on the time aspects of this change basically
if it has happened suddenly or progressively if it's a one-time thing meaning that the average ETA
suddenly increased on a particular day then there are a few factors i will look into
first i will check if we change the algorithm to calculate or predict ETA i mean ETA is an
estimated value not the actual time of arrival so i assume it is predicted based on an algorithm
if we change the algorithm to calculate it it may cause the increase of ETA's second
i will check if we change the algorithm to match riders with drivers as you mentioned earlier
it's the time it takes from a request is sent to the rider is picked up so if we have changed the
matching algorithm it's possible that it changes ETA's finally i will look into if we modified
anything in the data collection process anything that has changed the data may lead to the change
in the prediction so those are a few things i would check if the average ETA goes up suddenly
but if it has happened progressively it means that the train has been going up for multiple days
then i'd look at different aspects first i will look at the historical trend and see if it
follows a weekly pattern you know it's possible that one day of the week has higher ETA's
than others but if that's not the case and the average ETA turns out to be increasing steadily
for multiple weeks i would compare the ETA against the actual time of arrival i.e the ground truth
then we are facing with two possibilities one is that the change only happened to ETA's and not the
ground truth then we could further investigate if anything such as a recent product launch changed
the calculation of ETA's another possibility is that both the eta and the actual time of arrival
increase steadily for a while then there might be a big problem because it means that
it takes longer for the riders to get it right and it's possible that we will lose riders
if the trend continues in that case it's worth investigating the number of riders and drivers
if the number of riders increased or the number of drivers decreased both would indicate that
we don't have enough driver to serve riders then it takes longer for riders to get ride
therefore the average ETA goes up those are a few initial steps i would follow to diagnose the issue
would you like me to dive into any of the factors or to continue with more analysis
assuming none of the situations is happening let's assume that none of the situation is happening
how would you proceed to diagnose the issue sure if none of the about issue is happening
i would further segment the metric by different dimensions such as region and platform for region
i would check whether the change has happened in all geographic regions
or it was only happening in an isolated region if it only happened in a particular region
the problem might be related to specific things of that region such as traffic conditions for example
if it happened when the weather was not good then the bad traffic condition may have
caused the increase of ETA's or if there's a special or big event in a certain region
it's possible that the higher demand of that day has caused an increase in the average ETA
another factor i would check is whether the changes happen on our specific platform like iOS
android or web if so we could investigate if any bugs were introduced by a recent release
or feature launch on that platform both of these will help us narrow down the issue
does it make sense to you do you want me to do further analysis to diagnose the issue
or should i go ahead and summarize in my approach that sounds like a reasonable approach you could
go ahead summarize it okay in summary my approach to finding the cause of increasing average ETA is
as follows i started by understanding the data so that i could discard issues related to outliers
then i dug deeper to identify what factors could affect average ETA's to increase
suddenly or progressively then i look at when the change was regional or platform specific
let's now take some time to analyze the interview one thing you may have noticed is that the
interviewee did not follow the framework step by step after clarifying the question
the first factor she mentioned was related to the metric itself
if there's an outlier or not because an outlier would significantly change the metric
as i mentioned earlier if you can come up with something that's more specific to the question
itself it's much better than following the framework blindly to answer the question
another thing that's important is to interact with the interviewer for metric interviews
the interviewee will dominate the interview basically doing the talking most of the time
but it's always recommended to confirm with the interviewer whether your approach makes sense or
if the interviewer wants you to further clarify or dive into any of the factors you've mentioned
because metric questions are open-ended the interaction with interviewer could help convince
them that you are able to explain everything you come up with and your approach is robust
so there you go i have just shared one possible answer
to lift metric interview questions i hope it helps you with answering similar types of questions
if you see any interview question on the internet or you were asked in one of your interviews and
you feel it's pretty hard to provide a good answer let me know feel free to drop a comment below
make sure you are not breaking the NDA if it's from your own interview experience i'm gonna
provide guidance on how to answer each of them and i will select a few of them to make videos
to help you answer them better that is all for today's video stay tuned i will see you soon