In engineering and its various subdisciplines, acceptance testing is a test conducted to
determine if the requirements of a specification or contract are met. It may involve chemical
tests, physical tests, or performance tests. In systems engineering it may involve black-box
testing performed on a system prior to its delivery.
Software developers often distinguish acceptance testing by the system provider from acceptance
testing by the customer prior to accepting transfer of ownership. In the case of software,
acceptance testing performed by the customer is known as user acceptance testing, end-user
testing, site testing, or field testing. A smoke test is used as an acceptance test
prior to introducing a build to the main testing process.
Overview Testing generally involves running a suite
of tests on the completed system. Each individual test, known as a case, exercises a particular
operating condition of the user's environment or feature of the system, and will result
in a pass or fail outcome. There is generally no degree of success or failure. The test
environment is usually designed to be identical, or as close as possible, to the anticipated
user's environment, including extremes of such. These test cases must each be accompanied
by test case input data and/or a formal description of the operational activities to be performed.
The intentions are to thoroughly elucidate the specific test case and description of
the expected results. Acceptance tests/criteria are usually created
by business customers and expressed in a business domain language. These are high-level tests
to verify the completeness of a user story or stories 'played' during any sprint/iteration.
These tests are created ideally through collaboration between business customers, business analysts,
testers, and developers. It's essential that these tests include both business logic tests
as well as UI validation elements. The business customers are the primary project stakeholder
of these tests. As the user stories pass their acceptance criteria, the business owners can
be reassured the developers are progressing in the right direction.
Acceptance test cards are ideally created during sprint planning or iteration planning
meeting, before development begins so that the developers have a clear idea of what to
develop. Sometimes acceptance tests may span multiple stories and there are different ways
to test them out during actual sprints. One popular technique is to mock external interfaces
or data to mimic other stories which might not be played out during an iteration. A user
story is not considered complete until the acceptance tests have passed.
Process The acceptance test suite is run against the
supplied input data or using an acceptance test script to direct the testers. Then the
results obtained are compared with the expected results. If there is a correct match for every
case, the test suite is said to pass. If not, the system may either be rejected or accepted
on conditions previously agreed between the sponsor and the manufacturer.
The objective is to provide confidence that the delivered system meets the business requirements
of both sponsors and users. The acceptance phase may also act as the final quality gateway,
where any quality defects not previously detected may be uncovered.
A principal purpose of acceptance testing is that, once completed successfully, and
provided certain additional acceptance criteria are met, the sponsors will then sign off on
the system as satisfying the contract, and deliver final payment.
User acceptance testing User acceptance testing consists of a process
of verifying that a solution works for the user. It is not system testing, but rather
is there to ensure that the solution will work for the user i.e. test the user accepts
the solution. This testing should be undertaken by a subject-matter
expert, preferably the owner or client of the solution under test, and provide a summary
of the findings for confirmation to proceed after trial or review. In software development,
UAT as one of the final stages of a project often occurs before a client or customer accepts
the new system. Users of the system perform tests in line with what would occur in real
life scenarios. The UAT acts as a final verification of the
required business functionality and proper functioning of the system, emulating real-world
usage conditions on behalf of the paying client or a specific large customer. If the software
works as required and without issues during normal use, one can reasonably extrapolate
the same level of stability in production. User tests, usually performed by clients or
by end-users, do not normally focus on identifying simple problems such as spelling errors and
cosmetic problems, nor showstopper defects, such as software crashes; testers and developers
previously identify and fix these issues during earlier unit testing, integration testing,
and system testing phases. In the industrial sector, a common UAT is
a factory acceptance test. This test takes place before installation of the concerned
equipment. Most of the time testers not only check if the equipment meets the pre-set specification,
but also if the equipment is fully functional. A FAT usually includes a check of completeness,
a verification against contractual requirements, a proof of functionality and a final inspection.
The results of these tests give confidence to the client(s) as to how the system will
perform in production. There may also be legal or contractual requirements for acceptance
of the system. Acceptance testing in Extreme Programming
Acceptance testing is a term used in agile software development methodologies, particularly
Extreme Programming, referring to the functional testing of a user story by the software development
team during the implementation phase. The customer specifies scenarios to test when
a user story has been correctly implemented. A story can have one or many acceptance tests,
whatever it takes to ensure the functionality works. Acceptance tests are black-box system
tests. Each acceptance test represents some expected result from the system. Customers
are responsible for verifying the correctness of the acceptance tests and reviewing test
scores to decide which failed tests are of highest priority. Acceptance tests are also
used as regression tests prior to a production release. A user story is not considered complete
until it has passed its acceptance tests. This means that new acceptance tests must
be created for each iteration or the development team will report zero progress.
Types of acceptance testing Typical types of acceptance testing include
the following User acceptance testing
This may include factory acceptance testing, i.e. the testing done by factory users before
the product or system is moved to its destination site, after which site acceptance testing
may be performed by the users at the site. Operational acceptance testing
Also known as operational readiness testing, this refers to the checking done to a system
to ensure that processes and procedures are in place to allow the system to be used and
maintained. This may include checks done to back-up facilities, procedures for disaster
recovery, training for end users, maintenance procedures, and security procedures.
Contract and regulation acceptance testing In contract acceptance testing, a system is
tested against acceptance criteria as documented in a contract, before the system is accepted.
In regulation acceptance testing, a system is tested to ensure it meets governmental,
legal and safety standards. Alpha and beta testing
Alpha testing takes place at developers' sites, and involves testing of the operational system
by internal staff, before it is released to external customers. Beta testing takes place
at customers' sites, and involves testing by a group of customers who use the system
at their own locations and provide feedback, before the system is released to other customers.
The latter is often called “field testing”. List of acceptance-testing frameworks
Cucumber, a behavior-driven development acceptance test framework
Behat, BDD acceptance framework for PHP Lettuce, BDD acceptance framework for Python
Fabasoft app.test for automated acceptance tests
Framework for Integrated Test FitNesse, a fork of Fit
iMacros ItsNat Java Ajax web framework with built-in,
server based, functional web testing capabilities. Mocha, a popular web acceptance test framework
based on Javascript and Node.js Ranorex
Robot Framework Selenium
Specification by example Watir
See also
Acceptance sampling Black box testing
Conference room pilot Development stage
Dynamic testing Grey box testing
Software testing System testing
Test-driven development Unit testing
White box testing References
Further reading Hambling, Brian; van Goethem, Pauline. User
Acceptance Testing: A Step by Step Guide. Swindon: BCS Learning and Development Ltd.
ISBN 978-1-78017-167-8. External links
Acceptance Test Engineering Guide by Microsoft patterns & practices
Article Using Customer Tests to Drive Development from Methods & Tools
Article Acceptance TDD Explained from Methods & Tools
Article User Acceptance Testing Challenges from Software Testing Help