=============================================================
Minutes TAB call (November 13, 2015)
=============================================================

Info
Time: 3pmET
Dial:
Host confcall: OASIS
US Toll Free: +1 641 715-3822
Chat room: http://webconf.soaphub.org/conf/room/tab

Agenda:

1) Roll call

2) Q&A with Laurent on 'Self Certification at OASIS'

Public link to draft: https://www.oasis-open.org/committees/document.php?document_id=56869&wg_abbrev=tab

3) Approval of minutes

4) Status of public reviews

5) Status of open action items

6) AOB

Minutes

1) Roll Call
Attending: Ashok, Chet, Jacques, Kevin, Patrick, Robin, and Laurent

2) Q&A with Laurent

We organized the conversation using the questions that Laurent posed in https://lists.oasis-open.org/archives/tab/201511/msg00064.html

-- "Do you have an example of a certification program done by an SDO that failed and why? ... So, of the obstacles you list, would you say cost is the hardest to overcome? Can you think of ways to overcome via an open source or community effort (crowdsourcing)?"

General observation that programs that failed seemed to do so because of a lack of people willing to commit time and effort. This was why evaluating whether TCs were up to developing and supporting a program is high on the list of recommendations. Noted how most TCs short change the conformance clauses section as it is. A testing package is an even bigger undertaking.

Funding may also be a limitation. Note that programs that get funding are generally tied to a specific standard that a set of industry players want to support.

-- "A contingent benefit is better standards due to timely implementation feedback, when the test artifacts supportive of self-certification are developed in time. ... Are you suggesting that the test suite and the standard might be concurrent and mutually beneficial efforts, and that the test suite might find its way into the new "open repo" facility? Is this also true of test assertions? Test suites? Test harnesses?"

Discussion around Jacques' statement "the benefit of working on testing and conformance in parallel is that you get a better product." Jacques noted that the CAMP TC produced the CAMP Test Assertions Committee Specification. It took significant time but when Openstack came back and asked "how do you test your standard?" the test assertions document came in very handy. Also asking how things could be tested while the drafting was underway helped improve the CAMP spec.

Laurent was surprised that if this is valuable why aren't our members doing more of it. Discussion about how the different mind set, the lack of available skills, and the push to "get something out fast" all work against taking the time to address testability. Noted that open source projects coming to look at standards maybe changing the dynamic. "Open source guys are more concerned with how you prove / test a standard."

Discussed the idea of finding a proof of concept project, possibly TOSCA or CTI.

Laurent noted that this takes the conversation in a new and broader direction. "We need to understand how to work with the open source community - and how to incorporate testing to improve our standards." Laurent would very much like to bring this up with TOSCA and CTI to see if we can get something going. "If we had a poster child of success it would be incredible."

Agreed to reconvene next Friday to continue the discussion.

Minutes respectfully submitted by Chet Ensign.

Chat log

jacques (Fujitsu): will be away from computer for 30mn. But on the phone bridge.

Chet: Thanks Jacques.
Chet: Today's agenda:
Chet: ---
1) Roll call

2) Q&A with Laurent on 'Self Certification at OASIS'

Public link to draft: https://www.oasis-open.org/committees/document.php?document_id=56869&wg_abbrev=tab

3) Approval of minutes

4) Status of public reviews

5) Status of open action items

6) AOB
Chet: Roll call: Kevin, Ashok, Jacques, Chet
Chet: Patrick, and special guest Laurent

anonymous morphed into Patrick

Chet: Robin joins
Chet: Most likely candidates for self-serve project: TOSCA and CTI
Chet: A asks what funding would be for? L: for developing the test suites and the test environment.
Chet: E.g. like KMIP and SNIA
Chet: Turning to Laurent's questions:
Chet: 1. "Standards Development Organizations are often reluctant to get into certification because the process can be difficult, political, expensive, and fraught with risk."

Do you have an example of a certification program done by an SDO that failed and why? I can think of a couple including one in OASIS. Interestingly when I first floated this idea a few years ago, the stone wall I ran into was the magical word: "liability". I was assured by peers that this issue was irrelevant. I then discovered that one of the main issues was cost. So, of the obstacles you list, would you say cost is the hardest to overcome? Can you think of ways to overcome via an open source or community effort (crowdsourcing)?
Chet: J: so funding for a test suite tied to a specific standard like OIX. then there is funding for a general, cross-TC program.
Chet: L: so a test suite for a standard that solved a recognized problem *might* be able to get funding. J: if a generic, reusable tool were created for a project like that, it could help build across many TCs
Chet: L: discussed in depth with Don T. A proposal can be worded in a way that allows general applications even though developed for a specific project. E.g. oasis proposes to get funding for a test suite for stix and taxii - but the proposal could well propose to build the management tools and program that would be more broadly usable.
Chet: Example of a pgm that failed? A: yes - was involved in W3C discussions that went nowhere bcz people quickly determined this was going to be a significant level of effort. Nobody was prepared to step up and the idea frittered away.
Chet: So not cost but effort.
Chet: J: +1 - they were talking about a large test initiative - 100s or 1000s of test cases - so they could not attract enough participants to commit the tiem
Chet: L: my example is saml testing in kantara. know a bit about the biz plan and how it failed. It started as the Liberty Alliance as an effort to take testing out of oasis and perform tests in kantara. running the tests was outsourced to an indian startup and that quickly became expensive. Board at Oasis have that as a vivid memory and look back on it as a disaster
Chet: L: like Don's idea that in this self-serve universe, self-cert may be a better way to go
Chet: J: interop testing - very intensive, time consuming and expensive. that's what drummond is doing well. testing can go on for weeks.
Chet: J: the certification testing is a better first step - find bugs etc first in cert testing before turning to interop testing
Chet: J: example - the CAMP Test Assertion document - was a worthwhile effort but took significant time
Chet: J: the other effort that failed was WS-I - it was kind of a hybrid of conformance/interop testing - the tools and program was well designed. But the # of vendors who took advantage was small - just the big vendors already on the TC.
Chet: Maybe also because the standard itself was not widely adopted.

jacques (Fujitsu): @Chet: yes, TCs are neglecting the "conformance" aspect. They are expert in their domain, but not in conformance procedures...

Chet: C: I think the big obstacle will be TC interest, commitment and ability. Testing / certifying are different skill sets. Finding ways to assist the TCs in doing this is the big challenge
Chet: J: +1 - most TCs want to roll out the standard as fast as possible and worry about conformance later.
Chet: 2. "A contingent benefit is better standards due to timely implementation feedback, when the test artifacts supportive of self-certification are developed in time."

Now that's a remarkable insight. Are you suggesting that the test suite and the standard might be concurrent and mutually beneficial efforts, and that the test suite might find its way into the new "open repo" facility? Is this also true of test assertions? Test suites? Test harnesses? {I realize you answer this later in the document, but I'm curious about the role of open repos in this case]

Would any of that contradict this statement? "A test suite is developed, either by the TC or a TC member. This party may remains the IP owner of this artifact, but as it is a deliverable of the TC, it contributes it to the TC under the TC IPR mode." [I think we need Jamie here and too the liberty to copy him].
Chet: J: there is definitely a benefit of working on testing / conformance in parallel you get a better product.
Chet: J: even just developing test assertions. that's not as big an effort. it is more than just repeating the normative statements - you have to describe how they would actually be implemented in practice. in other words, you have to write test assertions really thinking about how implementations will work
Chet: J: writing the camp test assertions was a good example - often went back and asked the TC "so how would we test that" and it results in improved standard
Chet: J: problem of some mbrs developing based on a standard but not willing to bring back their experience to the group
Chet: C: maybe if oasis were in a position to offer people with the skill set (e.g. we got some funding to put something together) we'd find more TCs interested in taking it up. Perhaps we work with TOSCA or CTI in a kind of skunk works / sandbox undertaking to ramp it up
Chet: L: amazed to discover that that this is so useful but our members aren't doing it
Chet: C: I found in experience in development was that programmers didn't think they needed testing until they started having test engineers help up front. maybe our members will find the same
Chet: P: not doing testing isn't part of the bottom line of developing standards - most members of TCs aren't that close to the adoption end of the standard development process and so they aren't going to see the value
Chet: P: if you don't do the testing, you have no way of knowing the cost/value for your standard. having a proof of concept / test case would be very valuable
Chet: J: in case of CAMP, Openstack came back and asked "how do you test your standard?" The test assertions came in very handy.
Chet: Open source guys are more concerned with how you prove / test a standard
Chet: TOSCA is now starting to pay a lot more attention to testing
Chet: L: wonder if HEAT / Openstack has asked TOSCA about how they test their standard?
Chet: L: one takeaway I see from a strategic standpoint is that this is another way that open source and standards development are coming together. The observation that test assertions was a help to Openstack is another example of how these efforts can converge
Chet: L: can we hold another call next Friday?
Chet: L: this takes the conversation in a new and broader direction
Chet: L: also the most interesting topic I'm dealing with at the moment
Chet: L: we need to understand how to work with the open source community - and how to incorporate testing to improve our standards
Chet: L: definitely feeling the urgency of bringing this up with TOSCA and CTI TCs to see if we can get something going. If we had a poster child of success it would be incredible

20151113 (last edited 2015-11-19 19:15:16 by chet.ensign)