Test architectures for distributed testing

Testing architectures

In the context of OSI standardization, ISO and ITU developed a methodology for testing implementations of standardized communication protocols (here is an overview). This includes possible testing architectures, a language for specifying test cases (called TTCN), and the definition of information required to characterize implementation properties that must be known for apply a standardized test suite.

The most important test architetures for protocol testing are the following:

  1. Local test architecture: The tester has direct access to all interfaces - upper interface and lower interface of the Implementation under Test (IUT)
  2. Distributed architecture: An "upper tester" interacts at the upper interface (directly), and a separate lower tester accesses the lower interface through the underlying communication medium.
  3. Remote architecture: There is only a lower tester (the upper interface is used by the next-higher system layer)


In the case of local testing, it is assume that there is a tester that interfaces at the same time the upper and lower interfaces. This is what we assumed in the previous chapter on state machine testing.

The test coordination, shown in the first two architectures, means that the upper tester (UT) and lower tester (LT) communicate with one another in order to determine in which order the inputs at the different interfaces are applied, and in which order the observed outputs appeared at the different interfaces. The determination of the order of output assumes a common real-time clock of both testers - which is often unrealistic for the distributed test architecture (but not a problem for local testing).

The remote test architecture does not test the interactions at the upper interface of the protocol implementation. This is clearly an incomplete testing approach. However, certain protocol definitions (e.g. the original specification of TCP) do not talk explicitly about the service provided by the protocol to the user - I think this is a flaw. In the remote test architecture, it is assumed that there is a "standard" user process interacting with the protocol implementation at the upper interface.

Test architecture for embedded testing (general case)

The following diagram shows a quite general situation where the IUT is accessible by the tester only through some intermediate functionality, called "context" in this diagram (this is discussed in the paper Fault models for testing in context ( A. Petrenko, N. Yevtushenko and G. v. Bochmann ), in Proc. IFIP symposium FORTE-PSTV'96, Formal Description Techniques IX, R. Gotzhein and J. Bredereke, Chapman and Hall, 1996, pp. 163-178). One assumes that the specification of the behavior of the context is known and that this context is correctly implemented. The interactions of the IUT are only visible indirectly, through the interactions of the tester with the context. The following questions arise:


Synchronizable test cases - definitions

In the following, we consider the distributed test method without test coordination. We follow the arguments of our paper on distributed testing (Test generation with respect to distributed interfaces ( G. Luo, R. Dssouli, G. v. Bochmann, P. Ventakaram and A. Ghedamsi ), Computer Standards and Interfaces 16 (1994), pp.119-132)

The first paper that talked about problems related to distributed testing was our paper of 1984 (Synchronization and specification issues in protocol testing ( B. Sarikaya and G. v. Bochmann ), IEEE Trans. on Comm., COM-32, No.4 (April 1984), pp. 389-395; russian translation: Express Information (overview of western publications), Information Transfer, 1985, no. 28), which included the following definitions:

Synchronizable sequence: A sequence to two transitions is synchronizable if the input for the second transition comes over an interface over which there was an input or output during the first transition.

Non-synchronizable transition: A transition t to a state s is non-synchronizable if for each transition t' leaving state s, the sequence t.t' is not synchronizable.

The above definitions have been generalized in our paper on distributed testing where an arbitrary number of interfaces is assumed and each transition may have several outputs, possibly over different interfaces. See example below.


Definition (synchronizable sequence for transitions with multiple outputs): A sequence of two transitions is synchronizable if the input for the second transition comes over an interface over which there was an input or output during the first transition.

Note: This definition reads the same as above for transitions with a single output, but the possibility of several outputs gives rise to many more possible faults. Individual outputs may not only be wrong, they may also appear at the wrong interface, or not occur at all, or there may be some additional outputs. This leads to problems of fault coverage (to be discussed below).

Deriving synchronizable test cases - fault coverage

Definition: A test case is synchronizable if all pairs of successive transitions invoked on the specification are synchronizable sequences.

When using the distributed testing architecture, one would like to use synchronizable test cases. Questions:

Created: November 20, 2014