Software integration tests & integration strategy

Both IEC 62304 and the FDA require integration tests.

1. What are integration tests?

2. Why are integration tests necessary?

a) Improve the quality of the software

Integration tests, like all software tests, are intended to improve the quality of the software by finding errors. To do this, they check whether the interfaces of two or more software components (also called software modules or software units) fit together.

These integration tests are necessary, although professional software developers have already tested the public methods of all components. Because component, module, or unit tests are not sufficient in order to find errors in the interaction of these components. This shows the following example very drastically:

b) Comply with regulatory requirements

Many standards require integration testing and set requirements for it. For example, IEC 62304 requires integration testing and a verification whether the selected integration test method is suitable.

IEC 62304 requirements for integration tests

IEC 62304 sets the following requirements for integration testing:

• The existence of an integration plan
• Integration testing according to the plan
• Documentation of the integration tests
• The initiation of the problem-solving process if errors are found.

Requirements of IEC 62304 for the evaluation of the integration test procedure

In addition, manufacturers must evaluate and ensure that the chosen integration test method is appropriate.

Since the 2015 version of IEC 62304, section 5.6.5 has been worded somewhat more comprehensibly, as the term "verify" has been replaced by "evaluate" and the term "correctness" by "adequacy".

It is, therefore, not a matter of reviewing the correct application of the test procedure but rather the suitability of the procedure.

What the authors of IEC 62304 mean by evaluation can also be derived from Annex B.5.6:

• The integration strategy is suitable (see below).
• The manufacturer has determined the correct and necessary test objectives, e.g., functional correctness, timing, and maintainability (for example, in terms of weak coupling).
• The test procedure is suitable; this includes the methods to specify the test cases (see below).
• The manufacturer has selected suitable tools, e. g. for creating stubs/mock objects to replace hardware and software components that are not (yet) available or are to be tested.

This evaluation of the test procedure is usually done by a review of the integration plan.

3. How to perform integration tests?

Step 1: Determine the objectives of the integration tests

The first step is to define the objectives of the integration tests. The taxonomy of ISO 25010 provides a good checklist for the completeness of these objectives.

The selection and prioritization of these objectives should be risk-based. For example, if the timing of the software is critical, it should be evaluated during integration testing.

Step 2: Determine the integration strategy

Determining the integration strategy is part of the test plan. Manufacturers should already determine the integration strategy as part of the software architecture. This determines the order in which the components are to be integrated. Typical representatives of integration strategies are:

• Bottom-up strategy, which is popular with developers and requires many test- drivers
• Top-down strategy, which allows an early impression of the final product and requires many mock objects
• Big-bang strategy, which does not actually mean piecemeal integration at all but rather testing the entire system

In the bottom-up strategy, testers first test the hardware or operating system-related layers and then the higher layers (e.g., UI). This means that in the course of integration testing, they add the next higher layer each time, reviewing whether the test objectives are met.

Step 3: Specify test cases

The specification of test cases is part of the test plan. In order to specify the test cases and, thus, the test data, manufacturers should use methods such as the black box methods. These help to systematically determine the test cases so that they find defects with a particularly high probability. This is what distinguishes good tests from bad ones.

The specification of the test cases also includes determining how the tests are to be carried out, documented, and their outputs evaluated.

The documentation also includes an evaluation of the test result. In the event of an error, this documentation serves as input for the problem-resolution process.

4. What should you pay attention to?

a) Selection of the persons involved

In order to specify and perform integration tests, both programming experience and competence in software testing are required.

Software developers who have undergone further training to become software testers have this competence.

b) Correct timing

Manufacturers should not start software integration testing only when programming is complete. Instead, it is advisable to test the interaction of the available components following the integration plan from an early stage. In this way, errors can be found earlier and eliminated more easily.

During the maintenance phase, manufacturers should also perform the integration tests as regression tests. This way, they ensure that the changes do not reintroduce bugs that have already been fixed.

c) Compliance with regulatory requirements

IEC 62304 requires manufacturers to review, as part of the software release, whether the software integration tests have been performed as specified in the design and development respectively integration test plan. This assumes that these plans exist.

5. Conclusion & summary

Integration tests are an important instrument of software quality assurance.

The Johner Institute recommends that these tests be performed as a separate step and not together with the software system tests.

The integration tests help prove that the system can be assembled piece by piece from the components. This, in turn, confirms the desired weak coupling of the components and, thus, of a good architecture.

An architecture that fails to meet the "weak coupling and strong cohesion" paradigm becomes an architectural debt for manufacturers that becomes increasingly difficult to resolve. The expenditure for this "debt reduction" is a multiple higher than that for consistently accomplished integration tests.