Does ISO/IEC TR 29119-6 on software testing help in agile projects too?
ISO/IEC TR 29119-6 aims to help meet the requirements of the ISO 29119 family of standards (in other words the other family members) in agile software projects too. This claim is evident right from the title: “Guidelines for the use of ISO/IEC/IEEE 29119 (all parts) in agile projects”.
You can find out whether the standard meets these claims and whether you should spend EUR 173.30 to buy ISO/IEC TR 29119-6 in this article.
1. Context and objective of ISO 29119-6
a) What it’s about
The ISO 29119 family of standards describes best practices in, as the title states, Software and Systems Engineering – Software Testing. These best practices are independent of the specific development model, but many of them are reminiscent of a V model-like approach.
The following questions are often asked in agile software development:
- If you follow the best practices of agile development, will you also comply with the specifications of the ISO 29119 family of standards?
- How do you use the specifications of this family of standards in agile development?
These considerations are at least indirectly relevant to developers of medical software too, because IEC 62304 refers to ISO 12207 (Systems and Software Engineering – Software Life Cycle Processes). ISO 12207 in turn references parts of ISO 29119.
b) What ISO 29119-6 aims to achieve
The aim of ISO 29119-6 was to answer the questions set out above. In order to do this, it wishes to
- map the agile concepts onto ISO 29119-2,
- explain how ISO 29119-2 can be adapted for agile development, and
- show how the templates of ISO 29119-3 can be used for test documentation in agile development.
Further information
The article on software testing provides an overview of the testing of medical software and makes reference to other relevant articles.
The article on agile software development in medical devices provides important practical tips.
2. Structure and content of ISO 29119-6
a) Chapter structure
The chapter structure of the standard is unusual. Except for the usual first three chapters (scope, normative references, definitions of terms), it only consists of a single main chapter.
Almost the entire normative text is omitted in chapter 4.2. In this chapter, the standard attempts to map a total of 62 “agile practices and artefacts” onto ISO 29119-2.
b) Chapter 4.2: Mapping the agile concepts onto ISO 29119-2
Where possible, this mapping is carried out in table form, as the following examples show:
Example 1: Acceptance criteria
Agile concept | ISO 29119-2: clause | Mapping details |
Acceptance criteria | 8.2.4.2 Create test model | Acceptance criteria would be created as a test model that lists atomic requirements to be met before a user story can be accepted as “done” |
Example 2: Burn-down and burn-up charts
The second example shows that even an agile concept can be covered by several chapters in ISO 29119-2:
Agile concept | ISO 29119-2: clause | Mapping details |
Burn-down and burn-up charts |
7.3.4.3 Monitor
| Testers contribute to burn-down and burn-up charts by charts executing test cases that verify whether each user story’s acceptance criteria have been met. This daily test execution progress would be tracked via the monitor (TMC2) activity, task a, which focuses specifically on collecting metrics to track progress towards completion. |
| 7.3.4.5 Report | The status of testing, including progress towards testing on each user story, would be reported via the report activity. |
Example 3: Epic
There are also agile concepts for which there is no mapping onto ISO 29119-2. In addition to Epic, these also include the concepts of “Empowered team” and “Pair programming”.
c) Annex B: Mapping of ISO 29119-2 onto the agile concepts
Annex B describes reverse mapping. This looks as follows:
Clause | Activity | Agile practices and artefacts covered in Clause 4 |
7 Test management process | 7.2.4.3 Organize test plan development | 4.2.3 Acceptance test-driven development 4.2.6 Behavior-driven development 4.2.11 Continuous delivery and development … |
3. Benefits of ISO 29119-6
a) There is mapping between agile concepts and ISO 29119
If anybody wants to know how concepts of agile development “map” with those of ISO 29119-2 (and vice versa), then ISO 29119-6 is helpful.
b) The standard gives examples of documents
The standard gives examples for the following documents:
- “Session sheet” with the specification of a software test case
- Defect report, which shows the acceptance criteria and the results of the test
- Daily Test Progress Report with burn-down chart and overview of open and completed tests and their results
- Test Completion Report.
4. Criticism of ISO 29119-6
a) Limited conceptional integrity
The overview of chapters and the overview of the 62 (!) “agile practices and artefacts” in chapter 4.1 illustrate the problem: This standard lumps everything together:
- Best practices
- Activities
- Concepts
- Documents
- Roles
This does not simply result in a confusingly long list, it also makes the mapping more difficult because only the documents are relevant for mapping to ISO 29119-3. This approach also leads to there being elements in this list that (cannot and) do not correspond to ISO 29119-2 and therefore unnecessarily expand the list.
Other standards such as ISO 15289 spent a great deal of energy in bringing order to the chaos of the field of “software life cycle information”. This desire to provide order cannot be found to the same extent in ISO 29119-6.
b) Lack of transparency…
… of the structure
The lack of structure and internal logic is also reflected in the chapter structure. Sorting 62 subchapters in alphabetical order is at least unusual. Standards such as ISO 9241 indicate that hierarchical information structures should not contain more than eight elements to be sufficiently user-friendly.
ISO 29119-6 aims to help readers to understand how ISO 29119 can be applied to agile development. This is evident from the introduction to ISO 29119-6. The mapping of the standard onto the agile concepts is not, addressed in the normative part of the regulation, however, but in the non-normative annex.
...the selection of elements
The criteria according to which the authors selected the “agile practices and artefacts” is not clear. The selection seems like a collection of buzzwords on the topic of “agile development”.
It is consequently difficult to determine how complete the criteria are. It is fundamentally impossible for them to be complete.
Conversely, it is surprising that the standard includes “feature toggles” of the agile concepts.
… of the mapping
It is also not clear how the authors came to the specific mapping and the “mapping details”. Whether a test model was created as part of the backlog management (“create test model”) definitely also depends on whether it is a product backlog or a spring backlog.
It is even less helpful when ISO 29119-6 attributes an agile concept such as “behavior-driven development” simply to “all clauses” of ISO 29119-2.
c) Limited benefit
It is not just unspecific mapping such as “all clauses” that limits the benefit:
one of the most common Use Cases of the standard could have been that an organization that is developing in an agile manner must show that they follow the usual Best Practices as set out for example in ISO 29119-2 despite their agility (as if that were a contradiction).
The standard in the annex does link the Best Practices of ISO 29119-2 to the “agile practices and artefacts”, but ISO 29119-6 does not answer the question of whether this meets the requirements of ISO 29119-2. This, however, is precisely the critical point, particularly in audit situations.
d) Risk of confusion
There is even a concern that the standard is detrimental in audit situations. For example, a specification of a test case as set out in Annex D is problematic:
a test case with no description of the preconditions (except the software version) and with no description of the acceptance criteria does not meet the requirements of other software standards (such as IEC 62304, but also the ISO 29119 family).
The mapping of the ISO 29119-2 onto agile concepts such as Scrum leads readers to assume that the two would be comparable and could be attributed to one another. Scrum is a model for projects: it’s about how to create, divide, process and improve tasks in a (development) project. Scrum does not make any statements on processes or methods of software development. Scrum does not say how software architecture should be created or how test cases should be derived.
5. Conclusion, summary
a) Thanks to the authors
Writing standards is a time-consuming and often thankless task. Thanks should therefore go to the authors first of all for doing the work of describing how ISO 29119-2 can be used in agile projects.
There is some dispute about how well this objective can be achieved by a mapping of ISO 29119-2 with agile concepts.
b) ISO 29119-6: a really helpful standard?
ISO 29119-6 can be used to justify to skeptics that classical best practices and agile concepts do not contradict one another – if these skeptics still exist.
ISO 29119-6 lacks evidence that the application of all of the agile concepts mentioned in the standard meets the requirements, for example of ISO 29119-2.
After reading the standard, readers would be at a loss as the standard does not offer instructions. It also does not live up to its claim to be understandable if the readers do not know all of the parts of ISO 29119 (well). Without this knowledge it is not possible to follow all of the statements made in ISO 29119-6.
c) To buy or not to buy?
If you have a limited budget and are deciding between ISO 29119-2 and standard ISO 29119-6 described in this article: buy ISO 29119-2.
The Johner Institute supports medical device manufacturers with the agile and legally compliant development of their software. This means manufacturers can achieve lean documentation, the successful passing of audits and authorizations and a safe and speedy marketing of their products. Get in touch (e.g. via the contact form or via email) and find out more.
Author:
Prof. Dr. Christian Johner
Back To Top
Privacy settings
We use cookies on our website. Some of them are essential, while others help us improve this website and your experience.