>>Yvette Nameth: So next we have Puneet talking about testing integration between Twitter
services when he is ready.
>>Puneet Khanduri: All right.
So October 8th last year, Twitter signup was broken for almost two days.
Stock was down 5%.
How many people remember that?
[ Laughter ] Yeah.
So for the folks who raised their hands, that actually never happened.
And this is the reason why it didn't happen.
[ Laughter ] Sorry.
[ Applause ] All right.
So I originally came from platform engineering and then started realizing that we had bigger
problems to solve and ended up spinning up a new team called frameworks.
And one of the first things we built was this tool.
And the problem we were trying to solve with this tool was that, you know, you just refactored
your code.
We were in the process of migrating over from a huge monolithic Ruby app to microservices
or interarchitecture.
So lots of opportunities to break things.
And at the end of a refactor, how do you know that you haven't broken something?
If you are an SRE, aside from liability engineer, and you are in dev ops, then you have an even
bigger problem because you have no understanding of the code that's about to be deployed and,
yet, you are in the position to make the call as to whether or not a build is good enough
to put out into production.
And you try to mitigate that risk by Canarying and doing all sorts of fun things.
But eventually you want to get to continuous deploy, and that ends up being a hard problem.
So our natural first line of defense comes out to be unit tests and component tests.
And I personally have to make a confession, what better audience than this one to make
this confession, that I hate writing tests.
[ Laughter ] And the reason is the following.
If you look at unit tests, it can often take you more time to write good unit tests than
writing the code that's being tested.
Let's say your method has five possible CodePaths and you are interested in 100% path coverage.
Then one test will get you 20% coverage.
You write five tests and you're done.
Great.
I mean, I can write five tests.
That's not a problem.
But five tests for every method, eh...
Then you get a bit more ambitious, and you say I want component test coverage.
In this example, I have five methods in my service.
So my component is a microservice, and the request path has five methods in it.
And each method has five independent CodePaths.
So this works out to about 15,000 independent CodePaths.
Who here has written 15,000 tests for anything?
[ Laughter ] Right?
Wow.!
I have a lot of respect for you, sir.
>>> Our team has billions of test cases.
>>Puneet Khanduri: Right, right.
Kudos to you guys.
The thing to be cognizant of here is that as the complexity of the system that you're
trying to test grows, the relative impact of every handwritten test is smaller and smaller.
If I take the example of one of our services in production and I use the production cluster
that's running that service just to run tests, it would take about a thousand years for that
cluster to finish running all the tests covering every single possible CodePaths.
Meanwhile, the life cycle of that software is barely a week because that's how long it
will be before the next version of the code will get deployed.
So we need to be cognizant of the impact that we can make as complexity grows.
And if you don't trust me, hopefully you trust this guy.
So what's our approach to automatically trying to find regressions and figuring out whether
we have broken something?
First, we start by saying if you look at a test, it's finding a way to trigger a behavior,
right, which is providing a test input.
And what better way to trigger the behavior that you're interested in than to use production
traffic for Microservices.
We started using production traffic to drive the behavior of undeployed code.
And that, basically, made sure that we were exercising the CodePaths that were eventually
going to be exercised in production.
We preferred this approach to randomized testing or other kinds of testing because with those
approaches, while you might get different kinds of coverage and whatnot, there are a
lot of data-driven things in your code which only happen to surface when you get the right
sequence of characters and whatnot that are easy to find in production traffic than randomly
generated strings and things like that.
So using production traffic to drive the behavior of unemployed code turned out to be a good
approach.
And then the second part is once you've had your interaction with the undeployed code,
then you want to assert that the behavior was correct.
So how do you do that?
And you make the assumption that the stuff that's currently deployed in production isn't
broken.
Sometimes that can be the wrong assumption.
But most of the times if production is on fire, you find out pretty soon.
And if it's not, then you can assume that the stuff in production is good.
But there is a problem.
You're using the same request.
You're sending it to the new code and to the old code, and then you're comparing the responses
that are coming back from these two deployments.
And in this response you might have a lot of fields, deeply nested fields.
And some of these fields might be inherently noisy.
So how do you deal with them, right?
Your noise could be, for example, from server-generated time stamps because the clock is never going
to exactly the same across two different machines.
It's not the software's fault.
It's just a different clock.
You could have random number generators and you may not have set the seed, in which case
again you have another source of non-determinism.
You might have live data for downstream services that you're using for both the environments
and if that downstream data is changing rapidly, then that can be yet another source of non-determinism
because you can have a write-in between the two reads that the two live and undeployed
versions do.
So here's the big idea with Diffy, where, you know, once you have production traffic
that you're driving through the Diffy proxy, it will multicast this traffic to not -- not
just to the old code and to the new code, but two versions of the old code.
So in this diagram, you see a candidate at the top.
This is the new code; right?
The code that needs testing.
The primary and secondary are two deployments of the exact same known good code, which also
happens to be deployed in production.
So all of these three instances are running in my isolated staging environment.
Then when the traffic hits these instances and we get the responses back, we do a comparison
between the candidate and primary.
So that's a comparison between the new code and the old code, which gives us a raw set
of differences; right?
And then this raw set of differences has a lot of noise in it; right?
And then by comparing to the -- by comparing the old code to itself, which is the comparison
that's happening here between primary and secondary, we isolate all the fields that
are inherently noisy.
So anything that shows up as noise here between primary and secondary gets subtracted out
from the raw differences, and you get a final filtered set of differences that allows you
to focus in on things that you should be looking at as opposed to, you know, trying to find
a needle in a haystack.
And with that, let me switch over to the demo.
So as I showed you in that diagram, we need three instances.
So the first one up here, running on port 9000 is my primary.
The second one here, running on port 9100, is my secondary.
And then my candidate is running on this port 9200.
Right?
So let me look at my Diffy configuration.
So candidate is running at 9200; primary, 9000; and secondary, 9100.
Great.
So let's run this.
Okay.
So I have my Diffy instance deployed now.
And not doing anything interesting.
So let's send some traffic to it.
And since I don't have a production traffic source, I'm just going to run a bunch of Perl
commands that I have in this script here.
Okay.
So we just sent a whole bunch of traffic to this Diffy instance.
It showed up in the logs here.
And then you can also see that traffic being multicast to all of these, you know, service
instances.
So let's go back to the Diffy instance.
Wow, 100% of the traffic failing.
That tells me that this end point is called JSON.
So let's go take a look at what these differences look like.
Okay.
Dates coming out to be different.
49 seconds, 50 seconds.
Okay.
What else is different?
Names are coming out to be different.
Interesting.
And then time stamps are coming out to be different.
So notice that it says that, you know, 100% failing, 100% noise.
So the 100% failing here is saying that 100% of the time, primary and candidate disagree
with each other.
And the 100% noise means that 100% of the time, primary and secondary disagree with
each other.
Now, in this sort of situation where 100% of the time, primary is disagreeing with secondary,
you can be fairly confident that it's not your code's fault; right?
Whereas up here, in the "name" field, 12% of the time, primary is disagreeing with candidate,
but primary and secondary are always agreeing with each other.
So in this situation, you can be fairly confident that it is your code's fault.
Right?
So using this primitive logic and some primitive statistical analysis, you can apply this auto
exclusion logic, and it automatically eliminates, you know, these other fields where the noise
threshold is fairly high.
And if we deep dive into one of these guys, we see that the words here in the responses,
the names in the -- name field in the responses are capitalized here, whereas it's lowercase
here.
And it's, you know, capitalized here and it's lowercased here.
So that's sort of a regression in my service.
And I can deep dive into the response by, you know, blowing this up, and it gives me
the full-blown request that was used to trigger this behavior, so I can reproduce it if I
want.
And it highlights, you know, these nice fields for me and tells me exactly what's different
across the two.
And the sort of diffing that's happening here is it's recursively parsing the data structure.
In this case, it happens to be JSON, but we can also parse thrift responses and HTML and
do this sort of analysis throughout the structure of the response.
That's my demo.
So at this point, we can switch over to the master deck again.
Great.
Any questions?
[ Applause ] >>Yvette Nameth: Can you hit top, please.
I don't know why it doesn't always stay there.
How can you handle situations where production traffic contains private user data?
How can you debug?
>>Puneet Khanduri: How do you handle situations where production traffic contains private
user data.
How can you debug?
So when production traffic contains private user data, for redirect requests, we're just
able to use production back-ends.
And the way we're sourcing traffic is by instrumenting our production cluster.
And so the reads are fine, you know, as long as they're backed with production back-ends.
Writes are a bit more complicated.
And that's work in progress.
I can talk a little bit about it.
So what we're doing there is, not only are we recording the requests in production, but
also all the interactions with the environment for that request.
And so as a result of that, what you end up doing is, in your test environment, you're
able to replicate or dynamically mock the behavior of the environment as it, you know,
was recorded in production.
So that allows you to decouple the component that's being tested completely from the environment,
and you can do this sort of testing in complete isolation without having any downstream dependencies.
>>Yvette Nameth: How do you tell Diffy to ignore expected changes, such as bug fixes?
>>Puneet Khanduri: Good question.
We don't, and very deliberately so.
The reason is that we experimented with that earlier on, and we found that giving people
the ability to, you know, write exclusion logic that, you know, you can ignore these
fields creates a blind spot.
Because those exclusions have a life cycle of their own.
And, you know, it creates a burden of maintenance on the service owners that, you know, they
have to actively maintain that list, so that, you know, in the next release, they don't
forget to, you know, retire the exclusions that went out in the previous release.
And so the way our automatic, you know, artifact promotion process is being built is that if
we don't find any regressions, we automatically promote the artifact.
But if we find any regressions, we require a human to sign off on those regressions and
say that, yes, these changes are fine to go into production.
But, you know, from Diffy's point of view, we don't, you know, give them the responsibility
to maintain a list of exclusions or anything like that.
>>Yvette Nameth: Cool.
We have time for one more.
How do you mock the data back-end such that the undeployed code doesn't affect it adversely,
while still maintaining the similarity between responses?
>>Puneet Khanduri: I'm not sure if I fully understand the question.
>>Yvette Nameth: Basically, if you've got a back-end that is part of the -- the system
under test and you know your front-end is taking requests -- if you're only really trying
to test that front-end and you don't care about some dependency that's giving you, like,
weather information that's also applied to the tweet, how do you make it so that that
isn't actually what Diffy's finding?
>>Puneet Khanduri: Right.
So just for, you know, clarity, in terms of the scope of Diffy, Diffy does not take ownership
of deploying your downstream dependencies.
And that story is left up to the user.
But what we do in production or what a lot of services do at Twitter is that we write
our custom back-ends or, like, mocks for back-ends for these particular reasons.
So, for example, if you want to replay writes, one of the things that we're able to do or
some services are able to do is write -- create a mock back-end which just returns the okay
200 response but actually throws the downstream write away.
So it's not actually persisting the write.
And it's not trying to propagate that change for the downstream.
It isolates it.
And that allows you to again keep your component decoupled from your downstream dependencies.
Happy to chat more about it afterwards.
>>Yvette Nameth: Thank you.
[ Applause ]