Testing of component-based systems


Nature of V&V activities

Methods and techniques

Informal V&V techniques : Reviews (always based on written documents)

Verification (another meaning of this word)

Verifying that the requirements are satisfied. Using methods of formal proofs, or exhausive analysis of all execution scenarios, e.g. by model checking

Testing (Dijkstra said: one cannot proof the absense of bugs, only the presence of bugs - - nevertheless, this approach is very useful in practice.

Black-box testing

In this form of testing, the test cases are selected without the knowledge of the implementation. The test cases are based on the specification (requirements) of the component being tested, also called System under Test (SUT). Black-box testing is sometimes called conformance testing, because it checks the conformance of the implementation with the requirements.

In the case of distributed systems, each component is first tested separately before one does integration testing. One aspect of the specification of the component is the communication protocol to be realized for the interactions with the other system components. The protocol specification is normally written at a relatively abstract level, leaving many details as implementation choices. Therefore, protocol testing is normally black-box testing, where the protocol specification is the reference for the selection of test cases.

Conformance testing in respect requirements in the form of an FSM model

Fault models and implementation assumptions

Output faults only

A given transition has an output fault if the implementation of this transition produces a different output than specified by the model. However, it is not so clear what implementation of this transition means. Example from paper on Wp testing method - the implementation of Figure 3 has more states than the specification - the transition with input a from state 1 corresponds to two transitions in that implementation: input a from state I0, and (if S0 is reached by the transition with input from state S2) input a from state I3.

spec impl inpl

One assumes that (a) the FSM model is minimal, and (b) the implementation has a bounded number of states, normally the same number as the specification (also minimal implementation).

An example of a test case that is complete for output faults: An input sequence that lets the specification go through all transitions (also called Transition Tour). Optimization problem: Find a minimal (with the shortest number of transitions) Transition Tour for a given FSM model. For the simple FSM below, such a test sequence could be ****. It would find any output fault under the assumption that there is no transfer fault (see below). Note: In general, some of the transitions must be executed several times for obtaining a Transition Tour.

simple FSM

Transition faults

A given transition has a transition fault if the implementation of this transition goes to a different next state than specified by the model.

Transition faults are in general much more difficult to check.

Example: Consider the specification of Figure 1 above. A test suite for the Wp method is given below. Note that for state S1 the IS = {c} may be chosen, which leads to a different test suite. In this case, each state has only a single identification sequence - that is, the UIO method applies. Also note that for this specification, there is a Distinguishing sequence, namely a.b.



Note: These methods guarantee complete test suites only if one assumes that the implementation has no more states than the specification (which is not satisfied by the example implementation of Figure 3 above). The methods can be extended to cope with a bounded number of additional states in the implementation, but this leads to much longer test cases.- - Example: Consider the modified Figure 3 below. This machine is equivalent to the specification. If a transition from state I3 to state I1 has an output fault, this would not be detected by the above methods (without the extension mentioned).


Conformance testing in respect to extended FSM models

The above FSM testing approaches ignore interaction parameters and additional state variables as found in extended state machine models. Since realistic system specifications often are extended FSM models, it is important to consider testing methods for extended state machine models. The basic idea in this direction is to combine the FSM testing methods (for the pure FSM aspects of the behavior) with testing methods developed for functional (black-box) testing of software systems. In particular, one can apply:

Slides of invited talk at ICTSS-2015

Created: November 12, 2014