Conformance, Interoperability and Portability Testing: Proposed Procedures and Practices

Date: 27 July 2012

Author: Jacques Durand

Technical Advisory Board (TAB) Work Product
If you have questions or comments, Email the TAB.

Who is this for?

This document is for spec editors and TC members focused on interoperability of TC specifications.

1. Objectives

This document proposes testing practices for OASIS standards and specifications. It addresses the following requirements and assumptions:

2. Definitions

3. Interoperability vs. Portability

The conformance targets of a specification (i.e. what is being specified) usually fall into three major categories:

These categories are not exclusive from each other: often a specification will address two or all three categories. For example, some specifications describe both a text document format, and also the expected behavior of a text processor for these documents. Also, in some cases it is not possible to clearly distinguish which category an implementation belongs to. A process definition (such as WS-BPEL, or ebBP) can be seen as both a representation of a state machine and of a data artifact: it specifies both an execution logic, and a representation syntax.

Some implementations may be seen as relevant to both : e.g. in a layered architecture, a software module may operate on top of a software platform. Both are Processors, so may be expected to interoperate (interoperability). But one may also consider the Module as an item that is portable from one platform implementation to the other, i.e. will entail an equivalent processing of the overall system in both cases.

4. Testing for Conformance, Interoperability and Portability

Every specification should be written so that it provides enough guidance for the definition of Conformance tests. Conformance testing serves a different purpose than Interoperability or Portability testing. While the notion of implementation conformance applies to all specifications, Interoperability is only relevant to some specifications, and so is Portability.

From a product viewpoint, these different forms of testing will achieve different goals:

These two forms of testing are complementary, and in no way can substitute to each other. More precisely:

In all cases, a well-defined conformance testing process will speed-up the deployment of conforming products and facilitate the adoption of a standard.

5. Testing Methodology

The following process is recommended:

Recommended Testing Practices

Starting with Testable Specifications

First thing first: specifications should be written in a way that is "test friendly". This means:

Also, see interoperability guidelines http://wiki.oasis-open.org/tab/InteropGuide for errors to avoid when writing specifications.

General Process for Conformance Testing:

NOTE: in the following, the term 'members' identifies some OASIS members, who do some testing work within the context of a TC - either the TC originator of the specification under test, or another TC focused on testing tasks.

General Process for Interoperability and Portability Testing:

Examples and additional material about some of the above steps are:

6. Testing and the TC Process

There are a number of areas in the TC process where testing should be considered.

NOTE: Generally, there is a trade-off between time-to-market and quality of a specification. Depending on the specification, interoperability may or may not take precedence over functional needs. Testing and preparation for Testing has a cost in time and efforts, that should be weighted by the TC. Consequently the optimal approach to testing may vary from one TC to the other, as well as the timing of testing efforts within the general standardization schedule. The following recommendations correspond to an ideal situation:

Because conformance testing should always be done before Interoperability testing, as a way to simplify and speed-up interoperability tests, it is recommended that conformance testing takes place before interoperability testing.

Finally it is worth noting that testing plays an important role in the adoption of a standard and product marketability, and testing and demo activities can take place after a standard has been adopted including outside of the TC or even OASIS.

Status of Testing Deliverables:

It is appropriate to have test assertions and test suites undergo public reviews, since not all vendors participate in the TC that defines them. This means such deliverables should be considered for Committee Note or Committee Specification status - but do not seem to justify an OASIS standard status.

The following options should be considered:

Appendix

What is in a Test Report

A test report (either conformance testing or interoperability testing) should contain:

1. Indentification of related test material:

2. Details on Test participants and event:

3. Test Results:

NOTE: Besides a test report, a by-product of interoperability testing may be a set of “implementation guidelines” or an “implementation profile” that addresses interoperability issues not covered by conformance to the specification. A specification may leave out details left as an implementation choice or a configuration choice, and yet necessary for full interoperability. In addition, small differences in interpreting a specification may elude conformance testing and yet lead two implementations to be non-interoperable or non-portable. This will help overcoming interoperability issues not covered by the specification, and also contribute to faster adoption of a standard.

TestingPolicy (last edited 2016-07-27 15:46:24 by chet.ensign)