>>Yvette Nameth: Next up, we have Yabin, from LinkedIn, to talk about mocking the Internet.
[ Applause ] >>Yabin Kang: Hello, everyone.
[ Applause ] I know, it's about time for lunch.
But let me use the ten minutes to give you something more interesting and may help in
your day-to-day work.
So a little bit about myself.
My name is Yabin Kang.
I'm from LinkedIn New York team.
I have been with LinkedIn for three years.
So today I'm going to talk about the topic Mock the Internet, which I will first introduce
the new mocking system that we are currently working on, which is called Flashback, and
also give you the overview of the mocking strategy that we have in LinkedIn.
So Flashback is a network-layer mocking system that is taking care of all of the outbound
traffic that goes on the Internet, where you really don't want to see, like, the third-party
downtime network issue or results constraints that break your test.
So that's why we created Flashback.
Before we jump into the details, let's talk a little bit about why we like mocking, not
only for unit tests, but also for service-level integration test.
So why mocking?
Imagine we have the LinkedIn profile page.
Hopefully, everybody in this room has a LinkedIn profile.
Otherwise, you miss a chance to connect to other 400 million professionals.
[ Laughter ] This (indiscernible).
No.
The Marketing team asked me to do.
Okay.
Back to the profile page.
So it requires more than 100 downstream dependency servers to render this page.
If you test -- and most likely your test, it just tests one tiny specification on this
page -- you don't really want to rely on all of these dependencies.
Right?
And things are getting worse if one of the dependencies is actually an external service
that -- on the Internet, which you don't have control.
So the best way to solve this problem is mock them, mock all the dependencies.
Then you have a good test.
So take a look at this high-level picture of LinkedIn ingestion service.
(indiscernible) ingestion service is built by our New York team, and it is a service
to grab content from the Internet to enrich your profile to be more attractive.
So besides the -- besides the typical three-tiered architecture on the left, you will see there
is a gateway service in the middle.
It is -- it is working as a proxy.
And here is a good opportunity for us to mock in this layer in order to test our ingestion
service results rely on the external providers.
And this is where the Flashback fits in.
So Flashback still acts as a proxy, handles the HTTP, HTTP has traffic in terms of request
and response and store them.
It is controlled by test framework, so you can start it, like record or play.
You can also change the mock data on the fly from your test case since we have the API.
Or, as you wish, you can manipulate the response in a way to test the different variation of
(indiscernible) case for your service.
So, basically, you have the full control of what's going to be mocked on the network.
So let me show you some code.
This is the implementation about a record and replay inside Flashback.
Let me -- Recording, we actually take the response returned from the external provider
and are pairing it with the request that came from the internal service and store them locally.
When we're playing, we load the local stored response according to the request that came
from the internal service and just return them back.
So there's -- there's not going to be anything -- any traffic that goes on the Internet.
Also, I want to show you again some of the mock data, the actual mock data.
You can see we have all the information about the request and response.
And like I mentioned, you can actually manipulate the response by modifying this file.
Like, we can change the standard code to something else, from 200 to 404, to indicate a not found
response.
Or you can modify the content of the AGT body to something else, anything you want.
In this way, you can test your service in a lot more scenarios.
Then you can see how strong, how reliable your service is.
So, okay, Flashback is a good solution for us to test our ingestion service without relying
on the external provider.
But when we step back and take a look at the whole picture of how Flashback works, it's
actually -- it seems a common solution, just doing the mocking, the network, the traffic.
Right?
If we can make the Flashback more generic to be essentially at the service or the system,
it can help in a lot of other place in the same manner, like internal service.
So LinkedIn internal service are all RAS files, which means you have a common library being
used by the service and all the clients.
So to -- adding to Flashback is just like a config change to a Flashback-enabled HTTP
client.
Then you can use that.
It's quite easy, and nothing needs to be changed inside your service.
It's also a good solution for data service.
As long as your service is going through HTTP protocol, then you can enable Flashback, and
it will take care of the mocking part, mocking the data from the data service.
Also, it's more interesting to talk about the mobile client testing.
So when we do mobile client testing, we actually need to launch the emulator, launch the app,
and deploy your service, and having the real traffic between the client and your service.
But instead of making all this live traffic, if we can set up the proxy inside the emulator
and point to the Flashback, and then you can take advantage of over the Flashback and the
proximity of network mocking.
So it's quite easy to set up the proxy inside emulator.
And it also works for real device.
And, again, there is nothing that needs to be changed inside of the app.
You can just use Flashback.
Okay.
I hope you guys have the idea of what we are trying to build, the Flashback.
It helps you in the area of doing the service-level integration test that requires a network layer
mocking.
And we are trying to build it in a way that could be open sourced.
And it will be a library for anyone who wants to use or anybody who even wants to make contribution
back.
Or you can, like, come and join us in LinkedIn --
[ Laughter ] -- and let's work together and mock 'n roll.
Thank you.
[ Applause ] >>Yvette Nameth: Thank you.
What's the difference between using dedicated Flashback service versus using mocking frameworks
at the class layer like Mockito, EasyMock, et cetera?
>>Yabin Kang: So the -- Flashback is still, like, network-layer mocking.
So for those unit test mocking is kind of mocked at the class layer.
So when we mock the network layer, so you actually have the real traffic goes out to
-- goes out from your service, so that that's verified, like, a lot more than, like, just
inside the class.
>>Yvette Nameth: How does Flashback compare to WireMock or other mock proxy tools?
>>Yabin Kang: That's a good question.
I knew someone would ask this question.
So WireMock is -- Sorry.
So Flashback was -- the idea for Flashback was for our ingestion service.
We built that as a proxy.
Then we realized we -- actually, there is, like, WireMock or Betamax.
There is a lot of other mocking -- network mocking stuff there.
But some of them is now doing, like, a proxy way.
Some of them is all handled at the https traffic.
So we end up with we wanted to build something ourself.
>>Yvette Nameth: How do you deal with post requests?
And how do you deal with URL GET with variable parameters, like time stamp, random session
ID, et cetera.
>>Yabin Kang: The post request is the same -- is still just a request.
It's just different method.
So we have the post.
Like in the mock data, we have the method tab for get or post.
To deal with the different parameter, like time stamp, random session.
Okay.
So this problem is -- we don't have this problem for our ingestion service testing, because
we -- most of our case is, like, static content -- static URL and content.
But we do -- have thought about this.
We can make some parameter or variables inside your URL, or you help to indicate, okay, this
part of data is, like, dynamic.
We don't care.
We just care something like the -- important is a get and what's the URL, what's the body,
the content.
Yeah.
>>Yvette Nameth: Is Flashback open source?
>>Yabin Kang: Yes, it is going to be open sourced.
>>Yvette Nameth: That one's a repeat.
How do you manage situations where third-party providers change their layout or schema?
>>Yabin Kang: Mm.
Okay.
That's another good question.
So, basically -- so for the case for our ingestion service, we -- we don't have the idea when
the third party will change their content; right?
So we keep running our test for using the mock data, which is making sure our service
is still working for at least the old data.
Then we have some live traffic to making sure that's going to be working for the -- for
the new data.
And also, since you have the mock data, you can theoretically check the data versus the
new data from the -- the third party.
Then you will know, like, is there any change?
Is there any schema change or some content change?
If there's a significant change, you can rerecord the stuff and do the test again.
>>Yvette Nameth: How does it handle https requests?
>>Yabin Kang: Okay.
So since you know we are using, like, the proxy way, (indiscernible) require we have
to have a man-in-the-middle proxy.
So we kind of implement that.
And that's the most challenge in Flashback, since most of our URLs is going through https
traffic.
So we spend a lot of time on, like, handling the https and doing the man in the middle
thing.
Yeah.
>>Yvette Nameth: How do you keep recorded data up-to-date?
>>Yabin Kang: This is sort of like if the content change around the third party.
So like I said, for whatever the mock data that you have, you should be able to test
-- to run your test.
But if the data changed, you need to have a mechanism to check if anything changed between
your old data and the new data from the third party.
So -- so, like I said, you need to periodically check to see.
>>Yvette Nameth: How do you ensure mocked tests don't get forgotten about when internal
service APIs change?
>>Yabin Kang: Okay.
That's -- that's sort of similar to the last question.
But I think it's more like asking about the internal service.
Since we are testing, if we are running that against internal service, so you have the
idea of what version you're using for your downstream service.
So, basically, you have a service -- you have version for your own service, and you have
another dependency service.
So when update your dependency service, you know something might be changed; right?
So in the Flashback, either you do rerecording every time there is a change for the downstream
version, or we want to have some, like, mechanism to say my test knows my service, and we know
we only care about some particular data, like I only care about the first name, last name.
I don't care about, like, maybe someday there is a new field that says title or something.
So I can have a way to verify, like, if the new service -- if I construct the data schema
for the new service, still have the last name, first name, then it's good, I can still use
that data, I can still test my service.
And it should not break my service, should not break my test.
And should -- there is nothing I should worry about for the downstream update.
>>Yvette Nameth: And one last question.
They keep moving.
Where's the new one?
How is Flashback different than LinkedIn's other product, Sephia?
>>Yabin Kang: So -- so the Flashback started with, like I mentioned, for the ingestion
service, the proxy.
All our other products is kind of doing either the mocking inside the service or inside the
class, which is -- you need -- you have to modify something inside your service or inside
your code, which this kind of give you the benefit of proxy, as long as you can still
have a proxy, then you're good to go.
>>Yvette Nameth: Great.
Thank you very much.
>>Yabin Kang: Thank you, guys.
[ Applause ]