Componentizing End-to-End Tests - Nicholas Boll
>> Mic check.
Great, this works.
Guys doing well?
Woo!
OK.
All right, so for our next speaker, we have Nicholas Boll, talking about componentizing
end to end tests.
Are we ready?
All right!
[applause]
>> All right, starting a little bit early.
Just waiting for the slides.
I think I'm hooked up.
OK, all right, I'm talking about componentizing end to end tests, so, this is my first time
being at a conference speaking, so -- AUDIENCE: Whoo!
[applause]
Thank you, so I'm going to start with some ice-breaker questions that will make me feel
better.
So who here has written web applications?
Looks like quite a bit.
All right.
Who's written unit tests?
. Yeah.
All right.
So who writes end to end tests using Cypress, Selenium, etc.?
OK, less people.
All right, so who thinks they're easy to write, debug or maintain?
Not very many hands.
OK, cool.
So this talk will help.
All right, so I'm going to put a statement out there that writing end to end tests is
hard.
So one thing is figuring out how to select something.
Do you use -- there's all kinds of different things?
Do you even control what they are going to be?
Race conditions, both in your applications and also maybe in your test runner.
Data can be hard to control.
And also UI changes tend to break tests.
I know that's for the last five years at a company, we had an offshore doing our end
to end tests for us, and they were -- they had an entire team just trying to keep those
running, so the tests were constantly breaking.
Now that I've postulated all of that, I'm Nicholas Boll, I'm a design system engineer
at Workday.
It's kind of what it sounds like.
I'm an engineer on the design system to Workday.
It's called Canvas.
It's open source in July.
So how do we break down our applications, because we have to build complex applications,
so how do we do it?
So this is just a screenshot that I took from Essentia's website.
It's a complex application of just like a dashboard, some menu items, and stuff.
So we probably try and visualize how we would break this down and we'll probably draw boxes
around that, like layout boxes.
We might even call these components.
And obviously this application has multiple widgets, those widgets have titles.
We can start to see some common themes.
I had some color coding so the menu has menu items and I've outlined that with purple and
widgets I've outlined with red and the titles I've outlined with yellow.
So you can see these patterns of repeating piece that is repeat over and over again,
and this big problem was broken down into small pieces and built up to make this complex
dashboard application.
So how would we target that email icon?
So maybe we wanted to click on it.
So that one right there.
So we might choose the first option I had, which was a selector.
This is an option, this is called Xpath.
Hopefully you don't use that.
It's very fragile.
You can see the numbers in brackets, like 3, that's the third div under the body and
there was a second div under that one and then they anchor tagged.
You can maybe use a class.
That seems a little more stable, maybe.
That was a problem we had with our offshore team.
They tried to figure what selector would be best.
Obviously Xpath tried too often, so they tried class names.
So another option might be like a data test ID.
That is something that we could control and it made it a little bit more obvious that
that was something specifically for testing so that developers didn't accidentally change
that too often.
But then the next question is where do these selectors go?
Where do you put them?
So one option is co-located with your source code.
That seems like a pretty obvious choice, I've got the import React statement at the top
and I've just name it data test ID email so now you can grab that selector and you can
match that icon.
You can see that that's matched to the ID inside that email component.
So some pros and cons.
So it's very easy at the unit testing level.
It's nice and co-located.
It's easy to tell where it is.
It could be bad for end to end testing, so I don't work for Cypress, but I've contributed
there, so they gave me a nice t-shirt so if you do this in a Cypress test the way they
do that is every test becomes its own application that it wraps up -- it injects itself into
your application's page, so what ends up happening is instead of just grabbing all of your test
code, it also is now grabbing the whole of React, so that can be bad.
It will slow down your test, because it has to compile and parse and all that the entire
application, not just test code.
So the next thing that we tried doing was doing a global selectors file where we just
listed out and tried to add names to all the selectors.
Now, this is a nice list because everything has test IDs, but we had complicated CSS selectors
before.
It was a little bit more obvious.
What was nice is this was in our source code so that we could at least see where it was,
whereas before it was in a different repository in a different technology that we didn't have
secure access to.
So we couldn't even see what selectors the offshore team chose.
So this was at least a little bit better.
We could import them all and at the bottom we just said, this is all my examples are
in Cypress, but this could work for any testing framework.
So we just got this, you just we just called it cy.get.
Pros and cons with that, it's easy to tell where everything is at once, but it's not
modular if you've done some Redux stuff and you have all of your reactions in one place,
that works pretty well until you get to a certain scale and then that file is just huge
and you accidentally add selectors multiple times, because the list is too long.
So you could have local selector files, so that's what we ended up moving to next.
So we had each area had its own kind of selector thing, so all the icons had a selector, and
then all of the like maybe cards had a selector, and then just in the index file we just exported
everything, so now we could just do name spacing so now I have a cy.get selectors is a little
bit cleaner.
So it was more modular, but then we quickly ran into an issue where are selectors powerful
enough?
We have complex applications so we have more and more complex interactions to do something
that we wanted to do.
It's pretty easy to describe in English, but it was harder to do in our implementation,
especially if it had to be repeated.
So how do you target an item in this to do and then check it off?
So I'm talking about this one down here.
So if theirs one, you might do, maybe you'd say that could be that Xpath or maybe you'd
get all the to-do items and pick the first one, but maybe we want to just grab it by
something that we can see within the application.
So we might want to grab it by that name.
Now, some of you might be alluding to the next part.
Could we just use -- oops, sorry, that took a little bit.
This is too complicated for our selector, so we had to look for something else.
So I said before, all our examples are in Cypress, but it could work in any framework.
So then it was routed into testing library.
That added adapters for a bunch of different things, including React, Vue, Marco, cypress
and others.
But it's on one of the listed as one of their adapters.
So I thought that was pretty cool because kind of along that same idea.
So this is if you are using the to-do, using the testing library with the Cypress adapter,
you would use query by text and just say upgrade to SSD hard disks and then would you click
on it and then hopefully it doesn't work with how Sentra interacts with its application,
but that's cool if it works with the library which is around text and what about if it
is a little more specific where it doesn't quite work like a modern application the DOM
would work.
So you'd need something a little bit more custom to get the right thing.
So I coined it as component helpers.
Or basically component helper functions or helper functions.
We had a few different names for it.
So they look kind of like this.
So the first thing I did is just created a function called getto dobynow, the only place
that selector is available is inside of the helper itself, so your test code doesn't deal
with it at all.
The next with is giving it a check box.
And the last one is getting we want to check that to-do item and you can see I'm starting
to compose some of those helpers within the same file together.
So a check to-do item is now getting that item by its name.
It's getting the check box out of it and then it's clicking on it.
So it kind of looks like this.
We want the first check that to do item and then we wanted to grab it and get the check
box and do an assertion on it and make sure that it's checked.
now, you notice there's no selectors in T it's kind of the whole thing around page objects
where you're not dealing with the underlying driver directly, you're working at a little
bit higher level of abstraction it's not exactly like English, but it's fairly easy to read
and understand what's going to happen.
This worked well for our PMs and our QA to be able to have some confidence that we were
doing the testing that we wanted to do and that we said we were doing.
So other components, obviously like a button probably doesn't need much for helpers, because
all you can really do is click on it and maybe assert on its text.
But there could be other things.
So components with portaled content.
Now, portal is kind of something that was coined by React.
I think in Angular 4 they call it transfusion, but it's basically calling content and projecting
it to another place in the portal tree.
So if you can imagine a modal is like a picture frame and you want to take this picture and
shove it into the frame.
So that's what I'd describe as portaled content.
Something that moves somewhere else.
Portal specifically is going to be something different in the DOM.
So it's not not going to be a direct child.
It's going to be basically put at the bottom of the DOM so it's not confined by the overflow
of the container of the target.
So modals will do that so that it's not clipped by whatever that target is contained in.
So that can be tricky, because it's not -- you can't just find t you can't just take the
target and do a dot-find.
But now it's going to be scoped to the document instead of that component.
So hierarchical menus, if we have multiple layers of a menu, you have to click on it
and then you have to find your item and click on it and it opens up another submenu and
then you want to click on it.
That can be more complex where you just want to describe the hierarchy of it to select
what you want.
Another one might be combo boxes.
We might have some complicated things around how you'd select things.
Cards, you might want to say I want a card of some domain type.
Like I worked in a cybersecurity company so we had things like alarm cards and case cards,
so we wanted to just select that card by its name, but we wanted to get the card and not
the text, so we could just have a component that would grab the card by whatever the name
contained or ID.
And then return that card and do other actions on it, like change the status.
Dashboard widgets, similar to the example that I had, and the biggest one at the company
I worked with was virtual lists.
We had virtual lists for everything, which definitely made testing things harder because
we couldn't just grab something in the DOM and then click on it, because it didn't exist
in the DOM, it was virtual.
So now I've got a demo of how I've done that using React select virtualize, I don't know
if anyone has ever used that before.
So this is kind of what it looks like.
This is with 8,000 items in it, so as I'm scrolling, it will actually keep going.
It's opening and closing pretty instantly because it's only rendering exactly what you
see.
You can just go on and you can scroll quickly and get to everything.
So I've got a simple spec here that just grabs it, so this is without any component helpers,
this is just we are going to grab it, we have to know what the selector is it going to be.
We have to know what the next thing is going to be, which in this case I want to open t
I want to find something that contains the number 5 and now I'm going to click on it
and I want to make an assertion that it still contains that 5.
So I have to know the implementation details of how this works.
So I've got the first test and this is what it looks like.
It should just work, it looks fine, so why would I need any helper for this?
Well, if the item I happen to want to be selecting is not on the first viewable page right now,
that's an implementation detail for how the component works.
So let's say I want to select the item number 100 so we'll just change this right here to
100.
Now, this should reload and it's going to sit here and wait and it will eventually fail,
because it can't actually click on an option that has the string 100 on it, because it's
down there.
Now, interestingly enough, if I refresh this, and manually scroll it down to the item of
100, it will pass.
So to movement people that just happens to be an implementation detail.
All I really want to do is select a specific item in this list, the fact that it's virtualized.
I don't really care about it.
I shouldn't really have to care about that difference.
But it went through and it did everything that you you would expect.
It grabbed this item by the selection.
It went ahead and clicked on it.
Then it found an item that contains 100, it clicked on it and then it ran an expectation
against that to make sure it contained this this dash 100 on there.
So I just looked at the implementation of React select virtualize, I'd never actually
used it before, so it just made those helpers.
So I made some way of getting the component so that you could just call t you just import
this function and then you just call it: Then the select was a little interesting.
So what I ended up doing was I first grabbed that element and I'd click on it, just to
open it, then I'd end up getting this selector here, which normally you never have to think
about.
But this is the overflow container that contains all the virtualization.
So what I'm doing is I'm scrolling that.
So I'm sending a scroll event, I'm telling that container to scroll a little bit and
doing an assertion on it to see if the item I wanted it is there and if it's not I'll
tell it to keep scrolling and I'm telling it to make sure that React fully fleshes to
the DOM before that's done.
So here's the implementation of scroll, too, so this component opens it, or this helper
opens it, it grabs that container, it tells it to scroll, too, and it gives it a nice
long timeout.
The default is 4 seconds and it might take us longer, so I set that to 10 seconds.
Then once I've found it then I'm going to go ahead and click on it.
So what the scroll tool does is it's an async function.
It turns into a match that returns an async function that contains that element and it
will continuously see if there's any node in this node list that contains that string
that you gave it and if it doesn't, if the length is 0, it will make sure that the DOM
has been properly flushed.
It will set the scroll top to be + 20.
But it's hard to see what's going on.
So after that, then it just calls scroll top, sets it to the new scroll top selection, and
when it does, it will return that element so it can be clicked on later.
So what that ends up looking like, so I'm going to select that item No. 100.
And now I'm using my nice array API so I'm calling a get select.
I'm using pipe, so if people have used Cypress before, pipe is a plugin that I've made.
It's similar to .then, and then I just pipe in that end selection that I imported and
I give it what string I want it to match and then I'm doing an assertion of that containing
this text string.
So now this is what it ends up looking like.
It's going to scroll through automatically on its own and find the thing.
The right item.
So now I don't actually care if it's virtualized or not.
If it happens to be viewable or not.
It will just run through.
Now when I talked about pipe, this is what pipe gives me.
It gives me a nice little item here.
It tells me what function is called as well as a before and after of that call.
So if you use a .then, it doesn't actually show you any nice things in the log here.
So I'll slow that down a little bit and we'll watch that.
It's kind of fun to watch when it's running so we'll slow it down to maybe like 8 pixels
at a time.
So you can it's scrolling a lot slower, but it's the same basic concept.
And then once it finds the item it will either time out because it can't find it or it will
scroll enough that it finds it, it will select it and then it will move on.
All right.
So component helpers help us abstract our implementation details.
Just like components are a contract for developing our UI, component helpers are a contract for
testing our UI.
So we've -- we ended up proving this over the course of two years, but the implementation
can be tested.
As long as our implementation, our component helpers are updated with the implementation,
nothing ended up breaking.
How we ended up doing it is our component library was pushed into our local NPM repository.
Our helpers were also pushed to that name repository under a like a /helpers.
So our CI was tied to testing out our helpers of our components so we so we could guarantee
that as long as our APIs did not change, the application test did not break.
If they did, our CI would catch it before it ever got to that point.
So what ended up happening is all of our application failures are because the application was breaking,
not because the components changed.
So component helpers can be composed.
So we ended up proving this out for -- we had a multiple component libraries, and the
leaf node, the lowest Level 1 of our design system, it had these component helpers for
the very, very basic low-level components, and those were consumed by higher-level component
libraries, and those were consumed by widget repositories.
And then eventually it was consumed by the application.
So each level created their own helper abstraction layer and it kind of built on top of it.
So when we had more within our dashboard example we had some of those widgets or those apps.
They had some complex interactions and we just had easier ways of dealing with them.
One of them was like an identity list and it was using this idea where it would scroll
through identities and if it expected there to be an anonymous identity risk above a certain
level, it made it easier to give it some risk that we wanted it to test against and then
grab it and then move on.
So here are some of the links that I had.
First one is design.workday.com.
There's the ReactDOM testing library.
I have a link to the Sencha dashboard that I had and then cypress.io, there's no timer
up here so I have no idea where I'm at.
AUDIENCE: 1:51.
>> And this talk is supposed to end -- I'm actually done.
I hope I didn't leave too much time left.
[laughter] [applause]
Everybody, we ended a bit early, if you go out to registration, we actually have swag
bags and this would be a good time to go get them.
And our next talk will be 2:15.