Did you know there is a new software testing standard for the world, ISO29119? It was produced jointly by the International Standards Organization (ISO) and the IEEE. The standard is a result of years of work by members of ten countries, with approval by members of over twenty countries, as well as inputs by experts within the IEEE. ISO29119 starts by stating: “The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed to set of standards for software testing that can be used by any organization when performing any form of software testing”1.
This is a grand statement with lofty goals. This article will introduce some of the basic concepts presented in the IEEE/ISO/IEC29119 family of standards and examine where they might be of use.
The standard recognizes that there are many types of software, organizations producing software and development methodologies e.g., information technology (IT), personal computers (PC), embedded, mobile, and scientific, Agile, waterfall, large and small production organizations. Factors such as these influence software testing as well as if a standard should or would be used. ISO29119 attempts to be “open ended” for all users, and avoids the use of the words “best practice”. There are clauses with the word “shall” for those users who wish to have compliance statements which they can reference. However, the standard does allow tailoring by users. Speaking for myself, I would expect most users of the standard to practice tailoring to allow its use in different contexts, but it is still possible that test projects may not need to follow a standard such as ISO29119.
As of 2013, three parts of this standard were released and a fourth Part is scheduled for release in 2014. Each Part can be used standalone without the other parts, although there are ties between each of the parts, such as processes in Part2 produce documents which are defined in Part3. Each Part contains definitions, concepts, approaches, supplemental information, and examples. The next sections of this paper will briefly outline each Part with figures and a brief explanation. However, readers should refer directly to the standard for more complete details.
ISO 29119-1 (Part 1)
Part 1 introduces basic test definitions and concepts which support the other parts of the series. Part 1 is informative (meaning there are no compliance statements in it) providing overviews of test approaches, methods, the basis, and other concepts for testing. A primary basis for ISO29119 is the concept of risk-based testing, but other types of test bases are recognized such as math-based, model-based and experience-based. Additionally, general software testing concepts are presented, including: the role of software testing in an organization; project context considerations; testing within different lifecycles; and test planning.
ISO 29119-2 (Part 2)
Part 2 is the heart of the series and covers the software testing processes at the organizational, test management and test execution levels. Part 2 is has a strong risk-based testing focus. Risk-based testing is an approach to strategizing and managing testing that allows testing to be prioritized and focused. As seen in the figure below, Part 2 includes planning, risk analysis, test design strategy, scheduling/staffing, and document production. Part 2 details these general areas into specific actions and outputs that can be assessed and audited (for those seeking compliance).
ISO 29119-3 (Part 3)
Part 3 defines an extensive set of test documents. It is based on and will replace IEEE 829 Software Test Documentation. The standard defines sixteen documents with outlines and content descriptions, however, most projects would not produce all of these documents, tailoring to only the documents of use to them. For example, a project wishing to contract with an outsourced testing company could use Part3 to specify the names and content of a subset of test documents they wish to receive from the outsource provider. This would simplify the contracting process and allow both parties to have a better understanding of expectations by using Part3 as a compliance document. The list below outlines the basic test documentation names:
- Organizational test documentation
- Test policy
- Test strategy
- Project test documentation
- Project test plan
- Test project completion report
- Test Level documentation
- Test plan
- Test specification
- Test results
- Anomaly reports
- Level test status report
- Test environment report
- Test level completion report
ISO 29119-4 (Part 4)
Part 4 is still under production and voting, but it will define common names of test techniques in functional (aka black box) and structural (aka white box) patterns of testing. Each technique is presented with details on defining test: conditions, coverage, and cases (inputs, environment, and output). Additionally, each technique includes a coverage measurement. Sixteen classic software test techniques should be defined in Part4 along with identification of another 16 areas of quality characteristic testing, e.g., security.
Where might this standard be of use (or not)?
This section addresses my personal opinion and does not represent ISO, IEEE or any other organization involved in creating this standard.
Many testers believe the test field is not mature enough for a standard. They argue that we are still learning basic test concepts and ideas having only been a practicing discipline for a little more than thirty years (based on when Glenford Myers published the “The Art of Software Testing). While this is a valid point, the world test industry needs to start some place and I feel this standard is the beginning. Other people will use such a testing standard by blindly saying a test project must comply universally with everything in ISO 29119. I would say that is a misuse as it avoids critical thinking and testers must always be critical thinkers.
It can be used by groups such as:
- Companies working in regulated industries
- Groups working internationally to aid in communications about test contracting
- Organizations looking for ideas to improve their test practices and
- Researchers and academics wishing to assess which ideas in the standard are “right” or “wrong” in efforts to improve the standard.
The standard must still undergo the rigors of the scientific and engineering communities. I fully expect aspects of the standard to change however, the baseline it provides is a starting point for such investigations.
As I look back at how many regulated and process-driven industries worked, I was unhappy with their treatment of software testing. Blindly producing documents, as Part3 defines, without consideration of aspects of test processes (Part2) or techniques (Part4) has resulted in bugs being needlessly released. I do not believe that ISO29119 will by itself stop bad software testing in areas such as regulated software, but I do believe it can be a step forward if the standard is applied by thinking testers who use it as a basis for improvement—no matter their industry.
There are places where I would not use ISO29119. For example, if I were testing app software for personal use, or if I were doing short-term consulting (under a day or two) where I was asked to test or evaluate a product using my experience base, and where I would probably do exploratory testing without documentation.
Summary
ISO/IEEE/IEC29119 software testing is a standard for the world. It is the result of many compromises and through voting by many individuals from many countries. It is a baseline, yet it must change as practitioners and researchers use it. I would not present it as perfect, but then nothing in the software world ever really is.
Reference
1. ISO/IEEE/IEC 29119 1-4 Software Testing Standard, ISO/IEEE
About the Author
Jon Hagar Jon Hagar is a systems-software tester consultant supporting software product integrity, verification, and validation with a specialization in embedded and mobile software systems. Jon has worked in software engineering, particularly testing for over thirty years. Projects he has supported include: control system (avionics and auto), spacecraft, mobile-smart devices, and ground systems (IT) as well as working attack testing in his book: Software Test Attacks to Break Mobile and Embedded Devices. Jon is lead editor/author on numerous international standards.