All right, everyone
My name is Navin Iyengar
like you said
I'm a product designer in Netflix where I've worked for many different many years and different parts of the the Netflix product
and so what I'm talking about that I'm talking about everything you used to sign up for Netflix or once you remember
Sign into Netflix pick something to watch watch it
Binge watch on to the next thing all those things are you know pieces that the team that I work on
Designs and develops in just by show of hands who here actually uses Netflix
Alright so a lot of you so then you know, Netflix is fun and entertaining and relaxing for you
But you might be surprised to know how
scientific and rigorous and serious
We are and our approach to the design and the development of the product
And so I'm going to go a little deeper on that today talk about how we think about experimentation, which really means a be testing
And then I'll go into how that's really affected the way we actually design and think about the product
but first
This is not working so
You
I'll keep talking while you go but first
I'm gonna start with this guy
This is Galileo Galilei. Who is you all know? He's a scientist, right?
16th century in Italy
Now during Galileo's time science hadn't changed in two thousand years
and in fact science was really
philosophy the ancient Greek philosophers had looked around at the world made observations and said this is how the world works and
nobody really questioned their ideas all through the Middle Ages all up until
Galileo's time now one of the things that the ancient Greek philosophers had said
Was that the heavier something is?
The faster it falls through the air
Now on the surface that seems intuitive, right?
If you see an apple fall off a tree it goes fast
If you see a leaf fall off the tree it sort of floats down slowly
Sure, that seems intuitive now Galileo had a theory that that wasn't true
But in order to prove it he designed an experiment. It was one of the first experiments
And one of the most famous experiments of all time. So what he did was
okay, I'm just gonna use the
The keyboard here what he did was that he?
He went to the top of the Leaning Tower of Pisa
And he took two stones with him. They're about the same size, but they were different weights
okay, so they were made out of different materials and
What he did was that he dropped them off the Leaning Tower and what do you suppose happen?
Well, they hit the ground at the same time. Not what you'd expect to happen
Right, if heavier objects fall faster than light object. And so Galileo had disproven in one moment
What had been considered a scientific fact for 2,000 years?
That's really the power of an experiment now as to why a feather fall slower than a stone
Galileo couldn't really prove that if you look in his notebooks at the tire and he said, you know
It's probably has something to do with the shape of the feather or the air pushing up on the feather
But there was no experiment. He could run where he had remove all the air from the Leaning Tower of Pisa, right?
now it's cool is
That as technology improves you can run more sophisticated experiments
So recently on this BBC 2 show they ran the true experiment in in very dramatic fashion
they got a huge rock and a huge feather and they got a huge vacuum chamber where they pulled out all the air and
They they ran the true experiment and what he suppose happen
Soiler there. They are going to hit the crowd at the same time
and so we really appreciate this idea at Netflix that instead of listening to
Conventional wisdom or what people tell you is the truth you go you go into the real world
You collect behavioral data and understand get to your own conclusions
And the only difference is that instead of looking at the world of physical objects in motion
We're looking at the world of our customers behavior and customer satisfaction and trying to learn more about how that works
And how do we do that?
well
we run experiments - and we found that this idea of thinking of our work as if it were a series of
Experiments really frees up our thinking it allows us to try more innovative ideas and crazy ideas and maybe even stupid ideas
but in a small scale contained way first to really understand what the effects are of what we're doing before we
invest in building that at scale for
125 million Netflix customers
Because we're trying to solve for something that you may be - at whatever app or website you're working on
Which is that we wit we have way more
Ideas than we could ever actually design and build and ship and so what?
Experimentation allows us to do is try just a little bit of a lot of ideas to see where we should invest further
now at the core of every
Experiment is this idea of the scientific method? You may remember from high school
I know it sounds fancy
but really all the scientific method is it's a systematic way to learn information about how the world works and not just any information but
Reliable information information that you can use to predict the future behavior of the world
and so we use the same scientific method a simplified version at least for our experiments - so we're starting with a
Hypothesis a hypothesis is just a statement of how you think the world works
stated in a way that you can prove it - or false and
Then you're running some experiment. This is you gathering data from real world behavior and is unbiased away as possible
and then there's some analysis or result phase where you're taking what you observed and
What your hypothesis wasn't trying to decide whether your hypothesis is true or false now
It's worth noting that both of those outcomes true or false are actually pretty positive
Because if you especially in a product development context if you disprove your hypothesis
Then you have just saved yourself. A lot of time. You might have designed and built and shipped something
That actually doesn't have a meaningful impact on your customers satisfaction
and if you prove your hypothesis true also great because you've just opened up a fruitful new area of
Exploration that can improve your customers experience
And so really what I'm talking about experimentation
I'm talking about a be testing a be testing is the only way to truly run an experiment in a product development and design
Context and be able to say with a high level of confidence that the changes you're making are improving a customer's experience
so when we run into an a/b test
We we have a 95% level of confidence
That what we're doing is the right thing for the cost of customers that still means that one out of twenty times
We might be wrong, but it's the highest you can really get
And so let me talk a little bit more deeply about how we a be testing Netflix because it's a little more sophisticated than
Galileo's experiment
We're running controlled experiments, which means there's something that can call the control experience, which is our systems in production as they are today
no change because we want to compare that against whatever we're going to do to improve it and
ISO mentioned we're always starting with a hypothesis
This is just an example but on Netflix we might say that personalizing the content displayed on the homepage will
Cause and more people to retain their Netflix subscription
So Netflix is a subscription business people choose to pay us month-to-month
and what's great about that is that that is a
Very highly correlated with what we think of as customer satisfaction
If you're satisfied with Netflix
you continue to pay us and it's also good for us because it's some stains our business and helps us make more content and make
the product better
We have that hypothesis, we're creating a series of variations now these are different
interpretations of what the hypothesis is if you think about what
Personalization means on a given page that could be the algorithm that drives it that could be how its visually presented
All sorts of things and ways to get back at personalization
And so what we're doing is actually thinking very broadly about all the different things that we could do
There's a lot of art and judgment that goes into what we think
We actually what has the most potential and what we actually want to design and build
And now this is us running our tests
So we are randomly putting different people as I sign up - for Netflix into different experiences
this could be millions of people per experience and
And letting them try out that version of the hypothesis and so it's important to note here
Is that when they open their app and they're in that experience?
They don't know about all the other experiences so they act in a really unbiased way
now the other thing about a/b testing that I think sometimes people criticize is
That they think it takes all the creativity out of design and that's not true at all in my experience
What happens is that once you are allowed to do all these different variations?
You can try really creative ideas that are very different than each other. This is not about little button colors or pixel level changes
and then at the end of some period of time
We're determining a winner of our test and that's really going back to these metrics
Like I said from customer satisfaction, the more satisfied people stayed with Netflix longer and they were in this experience
So once we have that winner that's rolled out to everyone. That's the new default experience of Netflix
And that's the new control experience for future tests. We want to run
so in that way
We're constantly
Iterating and changing and building and moving the Netflix product forward but only in directions that make sense because they improve the customer experience
Encapsulate that idea and what I've learned from it is that we think of product development as a series of
Experiments and I propose that that leads to stronger designs whether you can test or not now
I know not everyone in this room has access to a be testing you need a lot of engineering
Infrastructure, you need a lot of customers to do that
Well, what I would propose is that even just thinking about your design process in this more methodical way can lead to stronger designs
Now so far I've been talking about
Hypothesis, like that's an easy thing to get to
But in reality you may have more
hypotheses than you know what to do with and you don't know where to start or you may be too overwhelmed and have no
hypothesis at all
And so there are a lot of different tools we have in our tool belt that we combine with a/b testing to develop a hypothesis
So for example, we're running
Ethnography studies. This means we're going into people's homes. Maybe only talking to two to three people a day
We're going to different countries like India or Thailand
but we're really living with people in their environment trying to understand how they live their lives what some of their unmet needs are and
You know, of course, we're only talking to two or three people
So that's not representative of how everybody feels but it helps us get an idea of some of the nuances behind things
And then of course we're running focus groups and usability studies
This is where we're bringing people
Into an artificial environment and it might be more people five to ten people a day we could ask them questions
Understand why they do some of the things they do
Or at least what they say they do we can put prototypes in front of them and observe how they use them and ask them
What they think of the experience?
But again, this is not representative of what everybody thinks. It's just the five or ten people we brought in that day
And we can run surveys. This is even more people
This might be thousands or tens of thousands of people that we can ask questions to them online. Right and we're asking their opinion
so, of course
There's some bias in that but what we do know, is that the more people we ask the more
Representative they are of what the larger population thinks at the end of the day though. It's still someone's opinion
And of course
We also have data we have trends in our own data that we can see and use this is a more quantitative method
But it doesn't tell us what hypothesis we should test really all it tells us is how people currently use Netflix
So in this case, I could say with 100% certainty that more people watch Netflix on TVs than owns
For example, that doesn't really tell me what I should do
what is my hypothesis for how you're going to improve the customer experience now that you know that and
So if I would could draw a line between all the things I just talked about which are ways to really develop
generate and develop that hypothesis
Really a/b testing is the only way to run an experiment and validate one of those hypotheses
And so we've probably run
Tens of thousands of tests during my time and Netflix and the most interesting ones are always the ones that reveal these non-intuitive things
about human behavior
and the
the top line of that is really that what we've learned is that you need to observe what people
Do not what they say. And that's what a be testing in its unbiased way of observing people really allows us to do this
There's this quirk of human behavior that people say things but that doesn't really correlate very well with what they will actually do
in real life
And so as an example
Here's a survey that we've run with
potential customers from Netflix many times
We've asked them what one thing
Would you like to know more about before you sign up for Netflix any guesses on what this big red?
Sector is feel free to shout it out
I think I heard it somewhere it was content
People want to know what is the content you have on Netflix?
You might be surprised to know that when you go to sign up for Netflix today
Or at least in the past you weren't able to find out what our catalog was
We actually made you sign up for the free trial and then once you're in there
You could pick whatever you want to watch and watch it over those 30 days for free
and
So this idea that that's strange has definitely been percolating up in within Netflix
Every time a new designer joins the Netflix design team to say hey, why don't we do this?
And so we had this hypothesis, right?
The hypothesis is that by showing people what our catalog is before they sign up for Netflix
It'll actually make them more likely to sign up
So that's the hypothesis that we needed to test and we could have just gone straight to a/b testing there
But we wanted to understand again
Like I said develop that hypothesis
understand some of the nuances behind it to make our designs even that much more likely to win in an AV test and
So we did some qualitative research
this was our
non-member homepage at the time which pretty simple pre focused how to start your free month button some supporting points and
Then we built a series of prototypes. I'm just gonna show you one for purposes of the narrative here where we allowed people to
Scroll down from there and see all the titles that we had available
You could navigate over to any different genre. You can see what horror movies we had on Netflix all the while
You couldn't actually play anything. If you try to play something. You still have to sign up for the free trial
That's just how our our licensing model works
so now when we took this into qualitative studies
People said yes, this is exactly what I need to decide whether to sign up for Netflix
And so we thought we had a hit on our hands
But when we actually observed them in these qualitative studies standing behind the glass and seeing how they use the prototypes
We realized that they were getting a bit bogged down. They were getting into what we call internally shopping mode
So they were looking for a specific piece of content or title. That's said to them
Yeah, I want to watch this so I should sign up for Netflix
And so we're starting to believe that this might not be actually the best nonmember
customer experience
But the only way to go from believing that to knowing that was to run an experiment to run an a/b test
And so we did that
Actually rn5 a/b tests each time
We had the control experience, which was our our normal page without access to the content and we had variations
we allowed people to go explore our content in different ways and
Every time the winner was actually the control experience experience without any content in it, very non-intuitive
Right what the learning here that we found?
Was that that's not actually you know
Just a page of content is not an indicator of whether you should join Netflix or not when people join Netflix for a free trial
There's all these other things that we can't communicate on that page
Like the fact that you can pick from all these different titles and they start immediately
You can start watching on one device and pick up on another the high quality of the streaming all of those things are not able
to be communicated just on a page where you can see what titles are on Netflix and
So it's a really good learning for us. Of course. We wanted to listen to our customers. Our customers are still
asking us for something which is that they want to know what the content is that have a better idea before they sign up for
Netflix and
So through subsequent rounds of testing
This is actually our homepage today, and I'm sure it may change in the future as we continue testing
But what we have here is a non-interactive wall of content that shows up in the background here and what this is it's personalized
It's tailored to what region you're in what content is available?
So it gives people an idea of what's popular in the region
Likely something that is they recognize in here and lets them know that hey
You're going to get to watch all these things when you sign up
But it doesn't let them get bogged down and trying to find that every specific thing that's in our catalog
And so I challenge you to think about the same thing
What behavior can you observe if you can if you can a be test sure do that because that is the most
unbiased way to observe people but even what we saw in our qualitative sessions, is that when we ask someone
whether it's the right experience they may say one thing but if you actually observe how they use it and
Try to be as objective as you can you'll see that there might actually be a discrepancy between those two things
So if I could just encapsulate that approach again is that we think of product development
there's a series of experiments and
I propose that that leads to stronger designs whether you can test or not because you can just approach your design in a more methodical
That's my time thanks very much
You