A substantial share of projects within financial market infrastructures
involve software development activities based on Agile methodologies.
In this video we will explore the software methods
that support large-scale transformation initiatives
and business as usual operations in high-availability environments.
I am coming to you from Georgia, the home of Exactpro's biggest delivery centre.
Georgia has ranked 7th among 190 states
in the World Bank Doing Business 2020 ranking.
Information and communication technology
has always been one of the key emphasis areas for Georgia,
with much attention being paid to education, innovation
and expanding the IT infrastructure.
The first Exactpro office opened in Tbilisi in the fall of 2018.
It expanded fast
and now we have two co-located offices with the third one opening this summer.
Today, the branch employs almost
300 quality assurance engineers and developers,
both in senior and junior positions, as well as software engineers in test
a new job description for specialists working
with our next generation test automation framework th2.
As a leader in software testing and related software development,
Exactpro has successfully focused on promoting professional excellence
in Georgia, building and nurturing a local QA community,
creating GeoSTQB - a Georgian representation of the international
software testing qualifications board, establishing links with universities
and actively showing support for IT students
from across the country who study here.
Internationally, Exactpro works with exchanges and clearing houses,
the most mission-critical institutions in financial services.
We deliver our services to clients in more than 20 countries
on all continents, covering the most scalable, latency-sensitive
and reliability-sensitive systems across the globe in our industry.
We have been in business since 2009
and employ over 750 specialists.
Exactpro is headquartered in the United Kingdom,
we have our delivery centres in the US, Georgia, Sri Lanka and Lithuania
with representative offices in Armenia, Italy, United Arab Emirates,
Canada and Australia on the way.
Our narrow focus is software testing.
But, what is software testing?
Let me answer this question from our Tbilisi studio.
We can google it.
These are the first links that Google Scholar gives us.
The top links contain words like art, craft, research, etc.
In software testing, a craftsman’s approach is presented as a continuum
from art to craft to science to engineering.
Some go even further and extend it to data
journalism or claim it to be magic.
Lots of vendors insist that their software testing product
is turning software testing into engineering.
And what is the difference between scientific inquiry and engineering?
These two processes go hand-in-hand expanding human capabilities.
It’s not possible to achieve great engineering
without a solid scientific foundation.
And cutting-edge science requires
more advanced instruments and tools produced by engineers.
But there’s a fundamental difference there,
while engineering is turning an idea into a useful product, the scientific inquiry
is trying to understand a physical object and build its abstract model.
Again, what is software testing?
Software testing is exploring software with the intent of finding bugs.
Software testing is an empirical technical investigation conducted
to provide stakeholders with information about the quality of the system under test.
Software testing is an information service
dedicated to finding things that decrease the value of a product.
It shows the presence of bugs, not their absence.
So, how do we measure the value of testing?
There are 4 main principles of how you measure
the quality of data or the quality of information service.
They are: Accuracy, Relevance, Interpretability, Accessibility.
The last one is Accessibility.
Information, if not accessible,
is of no value to others.
Think about a crypto wallet without a password.
The value is there, but you are not able to extract it.
At the same time, if information is accessible,
but others are not able to interpret it
then they are not able to use it.
Software testing is a social process.
Now let’s look at the first two data quality characteristics:
Accuracy and Relevance.
Both are of great importance from a software testing perspective.
Accuracy stands for our ability to detect important defects
and relevance stands for our ability to provide relevant information,
to detect defects as early as possible.
There are several measurements of accuracy known from machine learning classifiers:
accuracy, precision, recall (also known as sensitivity) and F1-score.
Some companies use the percentage of rejected defects
as one of the major KPIs thus prioritising precision.
In high-stakes environments sensitivity
is way more important than precision or accuracy.
There’s a film about software testing in a high stakes environment.
It is called "Pentagon Wars"
and there’s also a song about software testing in any environment.
It is called "Ironic".
If you read the definition of what ‘ironic’ is,
you’ll see that it’s a definition of what software testing is about.
An incongruity between the actual and the expected.
Relevance of information varies depending on the context.
If you refer to the song, it’s ‘a black fly in your Chardonnay’ vs.
‘a death-row pardon two minutes too late’.
In different industries and various contexts the relevance of information may differ.
And the relevance is most important for the Agile context.
Let’s talk more about Agile.
In Exactpro, we rely on a peer-reviewed paper
that defines agility as responsiveness to change.
Agility is not following particular rituals, it is whether a person, a team,
a company, an industry or a country can be responsive to change.
And we know that currently in the world
the only thing that's constant is change and another constant thing
is the challenges that we face trying to respond to change.
There are multiple definitions of Agile Testing.
One of them states that it is a software testing practice
that follows the principles of agile software development.
But, as already mentioned,
software development and software testing are two different processes.
This definition would be similar to ‘Agile cooking is a cooking practice
that follows the principles of agile hunting.’
In the ISTQB Glossary we have a better definition that defines agile testing as any testing
performed within an agile project and agile development practices.
The seminal book on Agile testing by Gregory and Krispin
contains several references, but we were not able to extract
the precise definition of what agile testing is.
On the other hand, these books contain useful
and frequently referred to concepts of agile testing quadrants.
You can now see them on your screen.
In quadrant 1 there are things like Unit Tests,
in quadrant 2 we have BDD and specification by example,
in quadrant 3 we have exploratory testing
and in quadrant 4 we have things related to the technical aspects.
If we think about Unit Tests we will realise that, it is the definition of software.
Software is a collection of instructions and data that tell a computer how to work.
The computer’s hardware is able to execute the software.
Each instruction causes the computer to carry out an operation
moving data, carrying out a computation,
altering the control flow of instructions or manipulating hardware.
So, if you look at this definition and then compare it
to what Unit Tests are, you will see that they are software.
One more time, writing Unit Tests is not performing software testing,
writing Unit Tests is developing software because unit tests are software.
Now to the second quadrant, BDD is also a set of instructions, you collaboratively
define executable specifications, and it allows you to run it.
What exactly are you running when you launch BDD tests?
You are running software that performs instructions and modifies data.
On the left we have software
and the processes in Q1 and Q2 are called
building software or software development.
On the other hand, the processes on the right is “testing software”.
There is a very clearly defined boundary between these two activities
that have an antiparallel flow of information.
Build Software to Test Software.
In order to run testing in the agile context at scale,
it’s necessary to have tools that will cover the quadrants on the right - Q3 and Q4
and help with software testing within large-scale agile implementations.
Software testing relevance is crucial in an agile context.
So, how do we define whether our software testing activity
is relevant to the overall process?
It’s relevant when the information that it provides is relevant.
For example, if we find more defects that the development team can process,
it means that a number of them might have a decreased value.
Or if a software development team is capable of building more software
than we can study, research and identify defects in, it once again
shows that our information is less relevant to the rest of the team,
because we provide the information later than necessary.
When you test within large-scale agile limitation, it’s necessary to make sure
that both processes can run in parallel
at approximately the same pace.
And it requires engineering both in building software and in testing software.
Any large-scale successful implementation that we’ve seen so far
has the same features and there’s a very good book that describes
large-scale Agile Development transformation within Hewlett Packard.
It's considered to be a huge success by reducing overall development costs,
increasing the amount of innovation and achieving
the dream goals of every large-scale agile transformation.
Among other things, what they did is building a large number of test environments
where information is obtained at any stage.
There are some checks that are executed on commit,
there are checks that are executed in a larger environment
and there are large-scale regression libraries that constantly run.
One of the things that you need for optimising feedback loops
in complex large-scale systems, is to throw hardware at it.
Because you need to have test environments
where you'll be able to run small unit testing
and you’ll also need to have large environments
where you are able to cover the system as a whole.
And also, if the next version is available,
you have two options - you can stop executing large-scale
automated checks and start against a new version
or you can proceed and take the new version as well
and execute it in some other environment.
By running more tests in parallel we can obtain more information faster.
The same Case Study presents a success achieved in development effort re-allocation.
Overall development costs were decreased by 40%,
and the effort spent on innovation increased 8 times.
The diagram on the screen shows effort distribution
between various activities before and after the transformation.
You can see that on the right side we have 77%, instead of 100%.
The remaining 23% are spent on developing and improving automated checks.
If you drill down into this 23%,
3% went to test tools development, 10% went to scripting checks
and 10% went into the failure triage process.
You can sum it up: 5% from most testing
is now automated and 10% from failure triage
and compare it to the original 15% from the manual testing.
Another important thing about large-scale agile testing
is that you are buying time by using people.
Regardless of how good your automated test library is,
the fact that you are producing more information faster
means that you need more human brains to analyse it.
And of course, you need to have a process to use them efficiently.
There’s a concept of cross functional teams.
It is a natural concept for humans.
You go hunting in the morning and then you’re cooking it in the evening
and the responsibilities are shared by everyone.
This process is present in any other area of the natural economy,
which to a degree was a direct implementation of cross functional teams
several thousands of years ago before it appeared
as a concept in agile software development.
But it’s not how industry or large-scale research work in practice.
In order to have information flow relevant for large-scale agile development,
you need to have a process that is capable of generating
enough information to fuel software development, and it cannot go
without specialisation when you talk about large-scale implementation.
Software testing for large-scale agile development
is built on three pillars: you throw platforms to buy time,
you throw people to buy time, and you also look into the process.
The process that helps you understand the system that you are trying to develop
and make relevant decisions faster because some software testing
cannot provide information faster.
We teach a lot of software testers.
We have several hundred who are working on Exactpro projects.
We have many courses available on this very channel (like, share, subscribe),
but what is the main thing that we are trying to teach?
Often, there is a mantra, that quality is everyone’s responsibility.
And everyone means all software developers and all software testers.
Quality is the firm’s leadership responsibility that cannot be delegated or outsourced.
The senior managers and executives within the company
should be responsible for quality to fulfil their regulatory duties.
The leadership can allocate resources, buy hardware, hire and fire software developers,
but the main responsibility of a leader is to obtain and process information.
The management is responsible for enabling channels
that will provide them with information from several sources
that they can critically assess before making decisions.
The worst leadership failures come from inability to perform this core duty.
On the other hand, software testers also have responsibility.
The responsibility for the quality of their own work.
When we teach software testers, we use two very interesting books in this area,
one is Extreme Ownership and another is The Scout Mindset.
There are no bad teams, there are only bad managers
and we teach software testers that they are responsible for everything related to their work,
as well as responsible for finding defects.
If you are not able to do it, then there’s something wrong
with what you do and with the software testing process as a whole.
The scout mindset is a motivation
to see things as they are not as you wish them to be.
Software testers also need to have this attitude,
uncovering the things that decrease the value of the software.
Defects are valuable information and information is the most valuable commodity right now.