[Applause]
hello your réguel keeper knock I'll try
sir good morning my name is Jorge and
today I'm going to talk about testing in
production with JavaScript welcome two
years conf Budapest and thank you for
coming to my talk hidden slide doesn't
matter I promise the talk it's not gonna
be too heavy and let's get started so
quick about me my name is Jorge I'm an
engineer I'm from Spain and I'm
currently working in Bristol started
working on autonomous indoor navigation
new abs and money area baekho's fronts
and then got back to the cloud space for
years working at I start back called
bitNami that just got acquired by BM
words so kudos to my team and now
currently working at Dyson so I always
like to store the talks with a quick
poll to know how the audience is doing
on on this topic I want to know how many
of you know what javascript is okay
thank you you know raising the hands
over there are here for the city the
food the party and just a bit of
JavaScript okay that's fine
know what testing is we're raising the
bar know about the testing pyramid or
any other way of classifying the
different testing levels yeah most of
you thank you who has a production
environment again every one of you
amazing who is afraid of testing in
production who is not afraid of testing
in production good give this is coming
so are you testing your services in
production not so many but there are
quite a few that's great the final
preferred spaces two tops yes spaces
right
so this this talk about this talk is
about I'm gonna try to answer some
frequently asked questions about testing
and your production workloads not going
to be too technical and not going too
deep inside any JavaScript code and it's
all based on the experience I gained
working I dance at Dyson I've been
working at Dyson for almost two years
now and Anna has been great and we have
learned a lot about testing in
production so this is a quick summary of
where I'm going I'm going to answer why
do you need testing testing production
was the right testing level how can you
escape or noise no js' work help with
this how to avoid disrupting the real
users because if you're testing in
production you're gonna be part of your
users and you're going to be using
production systems how to keep how to
keep your tests out of the statistics
reports and out of metrics so when you
report your to your manager to hire the
business didn't get you results and then
why you should clean up and what happens
if you don't because that happened to us
so yeah let's dive in a bit why do you
need testing in production well that's a
good question by how most of you can
answer that but really what do I mean
why what do I mean by testing in
production when you deploy your
application it could be an application a
service could be a lambda whatever thing
you deploy to production and has so the
public has access to is you need to
guarantee it's continues to work over
the days and over the hours and it's not
just it passed the unit test it passed
the service level test or whatever and
then I deploy and I don't care about my
service the service is running in
production and if something happens to
it you want to get notified not at the
user but by some automated system so the
idea about this is to have some kind of
testing tool that runs on a scheduled
basis
so it's testing your service like it was
a real user and if something happens so
the tests fail it means that a real user
wasn't able to do one of the actions you
have defined for your production
services and and then you get a
notification so let's let's dive into it
a bit why really do you need a testing
you need testing in production so here's
a quick sample it might seem dumb but it
can happen so imagine you have a lambda
function that we change the current date
right and it doesn't cut it for the year
2038 problem which is when you are
storing your date in a sign it 32-bits
variable you've run out of space to
store the date and then you overflow
right on the 16th of january 2038
so yeah you do your deployment you write
unit tests for it issue which on a date
if you query for the date again you
should return a different number should
return a different date yeah do you know
unit tests are fine
do you deploy it and after a handful of
years is stuff failing and it's because
a it has dynamic input/output it depends
on the current time on how the users are
interacting with the service or what the
service is returning dynamically but
going to a more real situation imagine
one of your stateful services so it's a
service that store some state on your
users or in your database or on the
state of your system then suddenly by a
bug or whatever becomes inconsistent and
then starts failing for all the
customers then when you have is a horde
of customers chasing you angry customers
saying this is not working I pay for it
but you don't want that you want
something that it's not going to get
angry at you and it's a robot an
automated process that's gonna behave
like a user it's going to test your
things your services in production and
tell you when something bad happens so
you don't get the feedback
from the users and so if you compare
these two situations so imagine you are
in the first situation where you don't
have an automated test system in
production and then something bad
happens and all your customers come to
you saying this is no working and
claiming back the money it's gonna take
some time for the users to report to you
that I know is happening and they have
to go to your service helpdesk or log an
issue or log a ticket or send you an
email or whatever so it's slow you don't
know your service is failing until you
get the feedback from the users but if
you have an automated system that runs
every minute every 10 minute every 20
minutes every 30 minute you can
notifications about your system not
working like this and also if you're in
the first situation you're losing money
because then the more time you last in
detecting that your systems are not
working that's money that you are losing
and and also you're paying the support
the supporting a lot for dealing with
those things while you could have catch
the Europe beforehand and also the final
the final comparison is about the
company reputation so if you have all
the users coming back at you and saying
this is not working this is a high pay
for this blah blah blah your company
reputation is over the over the floor
and then if you have an automated system
that notifies you by the way of an issue
in your production systems
nobody needs to notice it and then you
fix it quickly and then customers are
happy so that's the comparison and why
do you need testing in production so
what really is the right level of
testing you want to perform in
production because you don't want to
mess with real data but you're using
real systems yeah so this is a the
testing pyramid well one of the many
testing permits are but let me focus
let me focus on this one because it's
one of the simplest that I found it
comes from Superman article in from
ultimate from their website ultimate
home automation Panda and at the bottom
you have unit test then integration test
or service level test however you want
to call them and then the end-to-end
test so what happens when you have a
unit test you want a unit test before
you deploy your application so you test
a single functions of your application
or your service and then you deploy your
application because you are confident
that your application is working fine
because you have unit tested it but then
when you deploy you perform some
integration tests or service level tests
so you're focusing on a single service
the single service is the minimum unit
after deployment for us the engineers
you deploy several services and so this
end-to-end test so this integration test
or survey level test for those single
units from the engineering side but what
the user sees is like the whole system
and the user has user intense user
actions or also called user journeys
they are testing things across all the
system across all your services so
that's the kind of level I'm referring
when I'm talking about testing in
production you want to become a real
user you want to be in the skin of a
user and try all the possible actions a
user can do to make sure your system
still works as expected
how can JavaScript or Nadia's help with
that well this is where my example comes
into into show so at the moment I'm
working at Tyson and we have more than 1
million connected machine performing
actions against the cloud and calling
our services calling our Web API or
lambdas blah blah blah and and also uses
all around the globe doing actions and
performing actions just like renaming
your robot from Dyson 360 i2 mega ii
9000 whatever and then what happens is
that
it's a big system it's a form of several
moving parts and I want to make sure
that users can change the name of the
robot and don't get an error right so I
need a tool that's fast that's easy to
run in production so when I want to
verify that all the user actions all the
user journeys are working in production
I run the tool like this docker
container whatever it's not difficult to
deploy I want it to be extendable also
so when I write new services I write new
web api's whatever I can add new
libraries or new files and having those
testing situations and also I would like
to have the ability to write behavior
driven development tests if you want to
call them like that so really what I
want to find what they a scenario for
real user is what real users gonna how I
really use is going to interact with my
systems and and then try to reproduce
that automated simulating those paid to
a user simulating everything but no
emulates in the cloud emulating my
services the solution that we come to a
place was a tool written in JavaScript
using nodejs and the main libraries that
we are using are koo kumba a Jas and the
request library was also a bit of the
user library and basically what we try
to define a lot of singular steps a lot
of single actions and then using koo
kumba areas and then putting all those
actions together into a group to form an
x scenario you're gonna see now and yeah
then using the request library just to
make requests over the Internet HTTP or
HTTPS requests and the SL library to
assert that the values we are getting
back are they the ones that we expect
this is an example of of the cucumber of
a scenario written in cucumber yes or
what our next scenario really I want to
test
that given an existing user with this
configuration and the user locks in and
the user changes the robot name to
whatever then the one the robot name has
changed I want to test that that's a
it's an user action I'm not testing one
service
I'm not test I'm not testing one
function I'm testing the use of uranium
testing the user intent and that's what
we want we want to try and do what the
users will be doing in in production
with our systems this is a very simple
schema of how it works
we have the users the users at the
moment interact with a smartphone
application that smartphone application
sends commands to the cloud and then the
cloud communicates the comments back to
the to the robot to the cleaning robot
for example and I was testing to and
Mason tested to a ki ATT has built well
users because there are no real users
have a world a smartphone app and
that's making the calls to the cloud and
back and it's also emulating the robot
so we are simulating all the points in
production except the cloud our services
in production does we want to behave
like users but if you test in production
basically you are stressing your
production systems you are becoming a
user and if you run your testing
production very fast or you do a lot of
actions you become like a million users
talking to your services at a given time
and you don't want that so you need to
keep a balance between what you are
testing in production and what you want
to assert you don't want it's good that
you are very close to the to the user
and to the real actions but you need to
to balance it right so for it as an
example of this when we were writing the
test and
we started doing tests like every five
minutes I guess if I recall and what
happened is that we started getting
getting slowdowns on our web api because
we were quite in the way API so every
five minutes doing a lot of tests and
then it was a slowing down our
productions services and we don't want
we don't want that for the users and
also we experienced it
some DynamoDB throttling so database
throttling as I was testing was doing
cost to the API very fast and and today
and read and write to DynamoDB or the
database then we got throttle and that
was affecting users so you need to be
careful about that you need to a space a
bit your end-to-end test so the more
things you run them the faster you're
going to realize that you have an issue
if there's an issue but then you get
more noise because you get more noise
unless you get more noise on the matrix
you get motion more noise everywhere and
slowdowns
it slows down your production system and
so you want to keep a balance about that
and and just this is another another
quick bit when for this testing tool we
are simulating these users so we have
code in JavaScript how a user will
behave
what API schooled the user hit and what
does the user want to do like for
example changing the name of the robot
and yeah as I said we want to test a
real user behavior okay so basically
yeah again when you run your test in
production you are becoming another user
you are becoming part of the system and
equal skew your matrix for example the
number of users are interacting with
your system you want to report to your
manager or to the business how many
users are using your your production
services but if you have this end-to-end
test running in production then that's
giving your result also
if you want to know how many calls to an
API and also the error count so for
example you are very interested on how
many times your API switch on in a 500
or switch on in a 400 if you have the
end-to-end test that's gonna skew your
whistle so you need a way of keeping
those tests that you know about them
that you are generating them keeping
them out of the statistics and you could
also get mixed locks so if you have a
login utility that's put that puts all
the locks together like plank or
elasticsearch how do you discern which
locks were part of the end-to-end tests
or which locks were power of real users
doing real actions so yeah you need to
be careful with that how can we how can
we keep our test out of the stylistics
we could use a correlation ID so with
every request we would send as part of
the request a correlation ID or we will
generate a correlation ID that's going
to be pass between the different
services internally so every service if
they see the correlation idea starting
by end-to-end ffffff' they gonna know
it's part of the test they shouldn't
behave differently but at least not log
in matrix for example or log in matrix
to a different name space we could also
change the user alien in case of web api
s-- by yeah specifying instead of being
mozilla gecko whatever you have your
application and twin test and the
version and finally you can also add new
HTTP headers that nobody is gonna freak
about that you just put test to version
one point one point two or whatever and
then you treat them differently you not
treat them differently you don't lock
the metrics if you see the header coming
in into your production systems right so
after everything this this was a quick
introduction to how to run end-to-end
test and production
you now have you end-to-end test running
in production they are executing every
15 minutes every 20 minutes 30 minutes
and they are passing that's great but
what about all the fake that they have
generated so they they were built well
users or built on machines built or
smartphone apps that were creating new
users that were creating new cleans new
new data to our production systems and
that's fake data with them really wanted
into in the system and it's making some
it's it's it's going to cause a problem
in the long run so for example we were
login when you register a machine into
the cloud into the services we store a
row in the database saying this machine
is now just now six and belongs to this
user so as we were executing the
end-to-end test in production we were
creating new machines with new user
random machines random users bla bla bla
bla
we were filling the database and what
happened was that our services in
production were querying the database
and and getting all those users and the
services were expecting to find between
one and ninety and twist different
entries in production but we were
generating a lot of entries every single
day so the entries ended up like being
in the order of thousand entries what
happened that our service was not
prepared to do any pagination and then
it started failing for the customers
because we couldn't find their entries
because they were in their page six or
page seven because all the tests all the
testing production were generating more
data and filling up the database under
sub problems so yeah you need to clean
that data it's very important and you
want to do that automatically if
possible after every single test run
so it it helps to to keep your
environment clean it helps to start the
test the next test to start them from a
clean state avoid cluttering your your
databases or cluttering your services or
exhausting any of the limited resources
like IDs or yeah any any any other
limited resources you you could have in
your production system and and also
avoid as an example I just put about
it's making weird hm more difficult to
query to retrieve and to search so as a
recap testing in production is good and
necessary please do it if you are not
doing it you need to think as a real use
it to put yourself in in the skin of a
user and act like a user so that you
have really testing your your system
then using the framework cucumber yes
it's a good start and we found that it
work for us and it's it's amazing you
need to think also about your system
capacity so if your system cannot handle
a thousand requests per second then you
shouldn't be doing that many end to end
test space them a bit every half an hour
every 45 minutes whatever fits you you
also want to mark your test intents to
differentiate them from the from the
production load from the real users with
a correlation ID or with the HTTP header
or any other any other solution and
after that you want to clean your test
data and we set every connections or
every any status you might have changed
after after this test and the so
basically thank you for for listening
one more thing I would like to take a
bit boring job I will smile for the gram
selfie and then ever now because I have
plenty of spare time so now smile I hope
I can fit everyone
amazing and way
thank you but just wait are you brave
enough to test in production are you can
I hear you well yeah so now the applause
thank you
[Applause]