>> Good afternoon everyone, welcome to the third day of
Build conference, i hope you guys are having a good time
Looking at cool stuff happening across various products.
I am a program manager with the
Vsts test team and i'm going to talk about continuous and
Efficient and reliable testing with integrated reporting
Inside ci/cd.
I hope i will be able to show you features that we are
Bringing in that will help you in your development life cycle.
All right, so let's look at the key objectives and take-aways
That we want from the session. First we'll talk about, you
Know, some of the common challenges that we have heard
Around continuous testing and these are primarily based on,
You know, the customer engagements, community
Engagements and events like this.
We will then look at what vsts provides as capabilities to
Address these challenges and then we will give you a glimpse
Of what we are planning to bring in the future.
With that i will quickly talk about the areas that we play
Into. So our team wants the best
Platform which is the best platform used for authoring
Tests in various ids. It's across platform.
We then have support ci/cd platform and then we have
Rich reporting that comes out within vsts around the quality, basically, all right?
So let's look at some of the common challenges that we have
Heard around continuous testing. One of the things is about, you
Know, the lack of unified way of testing. When i say that, that means, you
Know, whether you are running your unit test or your
Functional test there is a lack of unified way to basically run
Tests. Then obviously there are tests
That take too long to run, so that's another big challenge
That we have heard teams saying tests take long to run.
There are tests which are important but then there is
Always flakeyness, especially when it comes to your functional
Testing area, right? it's difficult to debug and
Trouble shoot tests, and people are looking for easier ways to
Do that. Finally, you know, there is a
Lot of information that goes in when testing happens, so what
Are those insights that teams can get out of that that will
Help them to improve the quality process, right?
Those are some of the common challenges, i hope some of these
Resonate with you as well as. Let's look at what it provides
Us in capabilities. These are a set of features that
We have listed here but i'm going to group them into the
Topic of the session which is about continuous, efficient
Reliable testing with integrated reporting.
So let's talk about what "efficient" basically means.
Efficient can have multiple dimensions, like you can run
Tests in parallel to gain efficiencies, or you could run only the required tests in your
Pipeline. Right?
Similarly the reliability that i spoke about, tests being flakey
In nature, so how do we make sure that the tests re running
Have deterministic behavior or are at least able to identify
That some tests are flakey in nature? and then welcome to the inside
Part of it which is the whole look back which helps you get to
Know how your system is performing and what are the
Tests which are taking long to run or what are the tests that
Are failing more often, right? so that is where i'm going to
Switch over and show you some of the things, kind of a demo using
Vsds. Sow let's first talk about what
Unified testing means. Here i have a build definition
And if i expand that you will see the steps to find where i'm
Building a solution and then i'm running some unit tests.
Here i'm using visual studio task and then i have some tests
To find out here. This is all happening as part of
My build, right? but once i have run these unit
Tests in my build and my artifacts are generated i'm
Ready to deploy this into an environment, right?
Hear you see deploying to an environment and if i get into
Any q environment here, right, i have, you know, my app is
Getting deployed on azure, a web app, but i have tests being run
And these are functional tests, right, and both are using the
Same task, right? so you are not using different
Modes in terms of running your test or different platforms,
Basically there is unified way to run tests, right?
In here these are basically the functional tests and i will go
Into more details of this in a bit.
This is about having a unified way to run tests, right?
Let's come to about what efficient testing means.
So i've taken the same release definition here.
I'm going to cue a release for now.
Cue perform the release and
Let's go back to the definition where i want to show you options
In which, you know, you can run your tests in parallel,
Basically so the test provides you an option to be able to run
Tests in parallel if you go to the phase, look at the phase,
There is an option to say "multi agent" right?
These are the machines where your tests are basically
Running. You can choose that option and
Define the number of agents that you have.
Once you have defined those agents, your test recognizes
That i'm, you know, basically running in a multi-agent mode
And then you have options which are execution options and you
Can either run tests based on the number of tests, number of
Agents or based on, you know, prior execution times or based
On the assemblies, so you can do distribution based on any of
Those options, right? so if you go back to the release
That we had cued, by the way, what you're looking at is -- go
Back to release 34. You're looking at a pipeline and
That's the progress of the pipeline. You can
Get into that. If you go to the locks, you have
The set of agents where the tests are getting distributed,
Right? this is one way to distribute
Your tests for them to be able to run faster.
There are some more options that we have around distributing
Tests across, and we will be
Happy to talk about that so if you want to know more about it
Feel free to drop by. That's one part of it.
If you look at the other part, you know, you don't want to run
All the tests all the time. Sometimes you want to run tests
Which are impacted because of code change that you made, right?
That is, again, another option that we have on vs test which is
Run only impacted tests, right? so essentially there is a
Mapping between the code ha that be changed and the tests that
Are actually running. I will give you an example of
What that actually means. In hear you see a build
Definition where you had 115 tests and they all executed in
That particular run and so you had a run duration of about, you
Know, thirteen minutes out there. But the next time when we ran it
With only, you know, running the impacted tests, you see a large
Chunk of it which was not impacted, so it didn't run.
It only ran tests which were impacted, which impacted your
Total run duration. This is another way you can run
Tests efficiently which is using that flag, basically.
The third way is about, you know, when you are running tests
You want immediate feedback, right?
Yes, you can distribute, yes, you can run only impacted tests
But you don't want to wait for the entire pipeline to deplete
Before you can see the results and especially when there is a
Bad news. There is a failure of a test,
Right? so this is where we are bringing
In the in progress view for tests and now you don't have to
Wait for the pipeline to complete. While tests are executing you
Will be able to see the status and more importantly you can get
Into a particular test which is already executed and you can
Start looking at all the deeper, you know, error messages, other
Information, you can take actions,
Assign bots or change
The pass code itself. So that was the execution of the
Tests. Now let's come to reliability
Mrfrp if you go back to our pipeline, again, under the test
Task there is an option that we have brought in which is about
Read and field tests. Often tests are flakey,
Especially functional tests and what happens is you want to
Determine, you know, if the test is deterministic in terms of the
Outcome that it is deriving, right?
So what this option helps do is you can run your tests, specify
To read and fail tests and you can specify the number of
Attempts that you want to make for a failed test.
There is also an option to specify a threshold failure and
It's primarily that when you have a catastrophic failure and
You have a large number of tests that are failing you don't want
To do a read because the cost right?
So here we have specified a certain number of attempts and
What will happen is when you run these tests, you know, for the
Failed test it will be reattempted, basically. So if i go to the -- yeah.
So in this scenario if you look at it, i had 111 tests, i had 12
Tests which passed but 6 of them passed on read and that's an
Indication that these tests are not non deterministic, so i can
Go to the filters and select pass on read and it will show me
These tests. Can we remove the failed filter?
These have passed on read and something you want to look at,
Is it because of a bad test or for any other reason.
As of today you see only the last attempt but what we are
Trying to bring in if you go to the next window, very soon we
Will have a hierarchal type of
View where you will be able to see the attempts and the
Information associated with that test so if the failure -- if,
You know, the test passed for a certain reason you would have
Information for that but more importantly if there are
Multiple failures that happen for different reasons you will
Be able to get into each attempt and be able to see that
Information right there. So that's something that helps around the reliability of the
Test. The other thing is about rich
Reporting, obviously we have seen some aspects of it but this
Is kind of the new test that we are trying to bring in.
It's not just a new ui but there is a lot of in context
Information that we are bringing in, some of you you saw like the
In-progress type of view and we have always heard problems that
The space is not enough for you to see error messages and as a
Developer you can drag, get it to full screen mode.
There is support for attachment, we are planning to bring support
For previewing images because a lot of functional tests do
Capture screen images and you will want to see them right
There, be able to download all attachments together.
Those are some of the improvements. We've already seen if you could expand this one, additional
Test. This test out there.
Additional test. So, you know, this will be
Available for read and also available for order test and data-driven test.
This is an example of a data-driven test so that's
Another improvement that we're trying to make out here.
So this reporting is about what's happening in the pipeline now.
In this particular instance of execution.
But what we are also providing now, what we are trying to bring
Now is something called test analytics.
What this is going to do is for a given build definition or
Release definition, whatever tests you are running you would
Be able to observe those tests over a period of time.
So while i've looked at transactional information which
Is specific to that particular build or lead execution, i am
Now looking over a period of time. So in this case what you see is
A build definition which had failures, over the last 14 days
And then i see spike between these two days.
So i'm interested to know what caused the spike, right?
So i can drill down based on some of the filters out there,
So i go to the second tab, and in here what i'm using is
Containers, so when i do that i see that these are primarily
Coming from these two. So now i know which are creating
The noise and the teams that are owning that so i can reach out
To them and, you know, kind of drive accountability as well.
More importantly i can also get to individual tests.
So what you see here is the tests which have been failing more frequent and i can get into
One specific test. What you see here is an instance
Of one specific test that has been failing the most.
Then i can see all of the instances for that test for the
Last 14 days, right? and in the same view, i can
Select that test and see the details for that test for that
Particular duration, right? so the whole idea is that you
Can run tests faster and then you can get rich insights of
That particular execution and the system can provide you
Insights using analytics as well.
Those are some of the things, you know, that are coming up on
The test front. There are few more things that i
Want to call out. Let's go to the last one.
Last one. Yeah, so there is a new task
Which is vs test platform nailer which eliminates the need to
Have visual studio installed on the machine and you can use vs
Platform installer to acquire the test platform.
That's one. The second one is we have, you
Know, added a marketplace extension for screen resolution
For functional test which you can basically use to set the
Screen size. The third one is about support
For running tests on hosted agents so you can new run your
Ui tests on a hosted agent as well, right?
Those are some of the things that are coming very soon.
I would also like to give you a view of what's ahead.
Very quickly. These are the things that he
Saw. When we look ahead we are
Looking at the tests and those
Of you that have seen releases, we are bringing in the test so
You can use quality as a git
When you move across environments. Any thought around that, you can
Have test as a git and then you could choose an assembly, what
The threshold looks like but these are initial thoughts just
To give you a glimpse of what it means.
Analytics is going to basically expand and the one that is
Coming up in a few sprints is about failing tests but we will
Eventually get to long-running tests as well so analytics will
Provide you with more api sets and code coverage is something
That we want to invest in and integrate things in your
Pipeline to shift quality of lift where you can have prs and
Other builds basically looking at this information as well to
Drive quality. So that's all that i had for
This session. If you have, you know, any
Questions, we have a booth right across. Please fill out the evaluation