ISO/IEC TR 29119-6 aims to help meet the requirements of the ISO 29119 family of standards (in other words the other family members) in agile software projects too. This claim is evident right from the title: “Guidelines for the use of ISO/IEC/IEEE 29119 (all parts) in agile projects”.
You can find out whether the standard meets these claims and whether you should spend EUR 173.30 to buy ISO/IEC TR 29119-6 in this article.
The ISO 29119 family of standards describes best practices in, as the title states, Software and Systems Engineering – Software Testing. These best practices are independent of the specific development model, but many of them are reminiscent of a V model-like approach.
The following questions are often asked in agile software development:
These considerations are at least indirectly relevant to developers of medical software too, because IEC 62304 refers to ISO 12207 (Systems and Software Engineering – Software Life Cycle Processes). ISO 12207 in turn references parts of ISO 29119.
The aim of ISO 29119-6 was to answer the questions set out above. In order to do this, it wishes to
The article on software testing provides an overview of the testing of medical software and makes reference to other relevant articles.
The article on agile software development in medical devices provides important practical tips.
The chapter structure of the standard is unusual. Except for the usual first three chapters (scope, normative references, definitions of terms), it only consists of a single main chapter.
Almost the entire normative text is omitted in chapter 4.2. In this chapter, the standard attempts to map a total of 62 “agile practices and artefacts” onto ISO 29119-2.
Where possible, this mapping is carried out in table form, as the following examples show:
ISO 29119-2: clause
22.214.171.124 Create test model
Acceptance criteria would be created as a test model that lists atomic requirements to be met before a user story can be accepted as “done”
The second example shows that even an agile concept can be covered by several chapters in ISO 29119-2:
ISO 29119-2: clause
Burn-down and burn-up charts
Testers contribute to burn-down and burn-up charts by charts executing test cases that verify whether each user story’s acceptance criteria have been met. This daily test execution progress would be tracked via the monitor (TMC2) activity, task a, which focuses specifically on collecting metrics to track progress towards completion.
The status of testing, including progress towards testing on each user story, would be reported via the report activity.
There are also agile concepts for which there is no mapping onto ISO 29119-2. In addition to Epic, these also include the concepts of “Empowered team” and “Pair programming”.
Annex B describes reverse mapping. This looks as follows:
Agile practices and artefacts covered in Clause 4
7 Test management process
126.96.36.199 Organize test plan development
4.2.3 Acceptance test-driven development
4.2.6 Behavior-driven development
4.2.11 Continuous delivery and development
If anybody wants to know how concepts of agile development “map” with those of ISO 29119-2 (and vice versa), then ISO 29119-6 is helpful.
The standard gives examples for the following documents:
The overview of chapters and the overview of the 62 (!) “agile practices and artefacts” in chapter 4.1 illustrate the problem: This standard lumps everything together:
This does not simply result in a confusingly long list, it also makes the mapping more difficult because only the documents are relevant for mapping to ISO 29119-3. This approach also leads to there being elements in this list that (cannot and) do not correspond to ISO 29119-2 and therefore unnecessarily expand the list.
Other standards such as ISO 15289 spent a great deal of energy in bringing order to the chaos of the field of “software life cycle information”. This desire to provide order cannot be found to the same extent in ISO 29119-6.
The lack of structure and internal logic is also reflected in the chapter structure. Sorting 62 subchapters in alphabetical order is at least unusual. Standards such as ISO 9241 indicate that hierarchical information structures should not contain more than eight elements to be sufficiently user-friendly.
ISO 29119-6 aims to help readers to understand how ISO 29119 can be applied to agile development. This is evident from the introduction to ISO 29119-6. The mapping of the standard onto the agile concepts is not, addressed in the normative part of the regulation, however, but in the non-normative annex.
The criteria according to which the authors selected the “agile practices and artefacts” is not clear. The selection seems like a collection of buzzwords on the topic of “agile development”.
It is consequently difficult to determine how complete the criteria are. It is fundamentally impossible for them to be complete.
Conversely, it is surprising that the standard includes “feature toggles” of the agile concepts.
It is also not clear how the authors came to the specific mapping and the “mapping details”. Whether a test model was created as part of the backlog management (“create test model”) definitely also depends on whether it is a product backlog or a spring backlog.
It is even less helpful when ISO 29119-6 attributes an agile concept such as “behavior-driven development” simply to “all clauses” of ISO 29119-2.
It is not just unspecific mapping such as “all clauses” that limits the benefit:
one of the most common Use Cases of the standard could have been that an organization that is developing in an agile manner must show that they follow the usual Best Practices as set out for example in ISO 29119-2 despite their agility (as if that were a contradiction).
The standard in the annex does link the Best Practices of ISO 29119-2 to the “agile practices and artefacts”, but ISO 29119-6 does not answer the question of whether this meets the requirements of ISO 29119-2. This, however, is precisely the critical point, particularly in audit situations.
There is even a concern that the standard is detrimental in audit situations. For example, a specification of a test case as set out in Annex D is problematic:
a test case with no description of the preconditions (except the software version) and with no description of the acceptance criteria does not meet the requirements of other software standards (such as IEC 62304, but also the ISO 29119 family).
The mapping of the ISO 29119-2 onto agile concepts such as Scrum leads readers to assume that the two would be comparable and could be attributed to one another. Scrum is a model for projects: it’s about how to create, divide, process and improve tasks in a (development) project. Scrum does not make any statements on processes or methods of software development. Scrum does not say how software architecture should be created or how test cases should be derived.
Writing standards is a time-consuming and often thankless task. Thanks should therefore go to the authors first of all for doing the work of describing how ISO 29119-2 can be used in agile projects.
There is some dispute about how well this objective can be achieved by a mapping of ISO 29119-2 with agile concepts.
ISO 29119-6 can be used to justify to skeptics that classical best practices and agile concepts do not contradict one another – if these skeptics still exist.
ISO 29119-6 lacks evidence that the application of all of the agile concepts mentioned in the standard meets the requirements, for example of ISO 29119-2.
After reading the standard, readers would be at a loss as the standard does not offer instructions. It also does not live up to its claim to be understandable if the readers do not know all of the parts of ISO 29119 (well). Without this knowledge it is not possible to follow all of the statements made in ISO 29119-6.
If you have a limited budget and are deciding between ISO 29119-2 and standard ISO 29119-6 described in this article: buy ISO 29119-2.
The Johner Institute supports medical device manufacturers with the agile and legally compliant development of their software. This means manufacturers can achieve lean documentation, the successful passing of audits and authorizations and a safe and speedy marketing of their products. Get in touch (e.g. via the contact form or via email) and find out more.