Proposed Testability Guidelines (TAB Approved)

This document has no formal standing in OASIS at this point in time. It has been written by the TAB in order to exchange ideas within OASIS.

References to External Work


PREAMBLE: Not every specification qualifies for the testing of implementations. The notion of testability introduced here concerns implementations (of specifications) the behavior or properties of which can be observed and manipulated - e.g. repeated. Such implementations are typically embodied as devices, software or artifacts that are designed according to the specification. It may be the case that the behavior or properties of such devices are not easily observed or manipulated, or that the specification implementation is a process or set of practices that have to be followed by preexisting entities or humans, and cannot easily be repeated, decomposed or observed while unfolding. In such cases testing may be impossible or may have little value for a high cost. More generally, testing has a cost and this cost has to be measured against the expected value of testing.

NOTE: These guidelines concern specifications that qualify for testing, by the nature of their implementations and by a favorable cost/benefit evaluation.

A specification should be written with at least two kinds of readers in mind:

The best way to address (2) while helping (1) is to add a "testing perspective" to a specification in the form of test assertions. These test assertions (TAs) provide a bridge between the specification narrative and a test suite. They also lead specification writers to pay attention to the "testability" of their specification, i.e. how easy it is to verify that an implementation is conforming to the normative requirements.

"Testability" is a relative notion that depends on a future testing environment and its restrictions. Some feature may be untestable with a test harness that only supports black-box testing, i.e. is only aware of inputs and outputs of the implementation unit, while it becomes testable in a context where it is possible to insert some monitoring component (e.g. a "shim") in the implementation under test. Specification authors may not want to be concerned with these considerations. Test assertion writers will natutally have to assume some form of test environment or will influence test suite writers in choosing one. Even if little is known about a future test environment when writing test assertions, a major benefit of test assertions is to provide a better understanding of what is expected from implementations in order to fulfill the requirements.

The test assertion definition and model below is inspired and abstracted from the work of the Test Assertions Guideline OASIS TC, in particular the first CD produced by this TC:

What is a Test Assertion ?

A test assertion (TA) is a testable or measurable expression for evaluating adherence of [part of] an implementation to a normative statement in a specification.

Test assertions sit between the narrative of the specification or of its conformance clauses, and concrete tests parts of a test suite that verifies conformance or otherwise.

What a Test Assertion is not:

Why write Test Assertions, What does it cost you and How to do it ?

Benefits for specification writers:

When developped concurrently with the specification, test assertions may help to provide for a tighter specification. Any ambiguities, contradictions and unspecified features or behaviors (e.g. error cases, "corner cases") can be noted as they become apparent during test assertion creation. If not developed by the specification authors, test assertions should be reviewed and approved by them. The quality and usability of the specification are improved, and the extra time spent writing these test assertions will pay off in shorter overall time-to-deployment.

Benefits for test suites developers:

Test assertions provide a starting point for writing conformance and interoperability test suites. They simplify the distribution of the test development effort between different groups: often, test suite writers are not expert in the specification to be tested, and need guidance. By interpreting specification statements in terms of test conditions, test assertions improve confidence in the resulting test and provide a basis for coverage analysis (estimating the extent to which the specification is tested).

Benefits for promoting a Standard:

Increasingly, charters for new standard committees include a testing section that lists the writing of test assertions and sometimes of a test suite, as parts of the deliverables of the new TC. The credibility of a standard is better when it is delivered along with a test assertions document or better with a conformance test suite. Users have more confidence in its maturity, and see a shorter path to implementation.

Benefits for Product Vendors:

Product vendors benefit from several angles when a set of test assertions is made available with a standard: they get a better understanding of implementation requirements, can use test assertions in their QA process, and have a conformance indicator that will help them reduce discrepancies with similar products from other vendors and reduce interoperability troubles for their customers. Overall, the time to market is shortened.

Constraints for specification writers:

Writing test assertions is not a negligeable effort. It takes time and commitment to develop a useful set of test assertions, which in turn may delay the completion of a specification. It is recommended to elect early on a "test wizard" in the specification committee, before the first specification draft is complete. The role of the test wizard will be:

An important part of this effort is to establish the local guidelines and practices for developing a consistent set of test assertions:

When to write Test Asserions:

Several options may be considered:

Overview of a Test Assertion

A test assertion (TA) includes:

In addition, a test assertion may optionally include:

An Example

Consider the following as a requirement from a specification on “widgets” :

Here is a test assertion addressing this requirement:

The TA predicate is worded as an assertion, not as a requirement: the 'MUST' keyword of the requirement is absent from the predicate but reflected in the prescription level. (In fact, the predicate would be the same whether the keyword is MUST, SHOULD or MAY). It has a clear Boolean value: either the statement is true, or it is false for a particular target. Additionally, in the above example "testability" depends on some properties of the testing environment, assumed here to only have the ability to detect the shape of surfaces - e.g. by optical scan. The wording of the predicate takes this knowledge into account, by restating the normative requirement in terms of "facets". Here the wording of the predicate may also provide guidance to a test suite writer who is not expert in 3D-geometry. As illustrated in our example, the predicate wording must not however modify the semantics of the referred normative source. A domain expert can ensure that the predicate is a good semantical match to the normative source.

NOTE: it is not always the case that such an equivalent rewording can be done when writing a test assertion predicate. There is still value in keeping the TA predicate "abstract" from test environment conditions.

Assume now that the specification requirement was:

There is now a pre-condition for the applicability of the previous test assertion. It will translate into an additional Prerequisite element:

The prerequisite could refer to another test assertion in case the property "made of plastic" is itself subject to testing, which would be the case if "plastic widget" is not a property known upfront.

Advanced Features

Test assertions may support more advanced usages that are described with more details in the Test Assertions Guidelines []. Among advanced features:

TestabilityGuide (last edited 2013-10-29 18:13:22 by jdurand2)