Interoperability Testing Framework¶

I've been thinking about this problem for awhile in the context of functional testing. In my group at Cisco a lot of our functional testing is not yet automated because it involves physical devices or complicated server configurations (and multiple servers). We've covered a lot of ground with unit testing, even to the point of a unit test setup which starts a service listener first. We've gotten some nice results with that, but it has its own problems. Mainly, it only works java-to-java and csharp-to-csharp. Plus with both service and client in the same process you get some non-standard interactions. And so I'm ready to take the next step and tackle the problem of automating a unit test which requires some additional server setups in other processes and languages.

A second issue is, once I have a setup, I want to be able to run it again and again with variations in arguments. This is hard to do within the unit testing framework itself.

A non-issue for now is automating tests which run on different hosts. So I don't want to do anything that disables that, but I'm not going to try to solve that problem right now. An example is a csharp client running on windows hitting a java server running on linux. So, good idea, we need that, but not today, not now.

Model¶

The model for an interoperability test is to run a test a number of times with different configurations. In a classic etch test, I might run the programs again and again with variations in the urls used to configure the listener and the client. For example, I might run the test with and without the Logger filter, or run the test with both tcp and tls transports.

interoptest ::= run*

run   ::= test "(" param* ")"

param ::= name "=" value  
name  ::= <string>  
value ::= <string>

The model for a test is setup, support, jig, and cleanup. Each of these is one or more programs which are run. First, the setup programs are run to completion, and perform any required initialization tasks (create required directories or files, init db, etc.). The support programs are then started. They will run for the duration of the test. The jig is a program which is then started and allowed to run to completion. This is best done as a unit test but doesn't have to be. When the jig is up, the support programs are stopped. Finally, the cleanup programs are run to completion, summarizing any test results and cleaning up the mess.

test    ::= setup* support* jig cleanup*

setup   ::= program "(" param* ")"
support ::= program "(" param* ")"
jig     ::= program "(" param* ")"
cleanup ::= program "(" param* ")"

The model for a program is a command line interface. This maps well to both windows and unix operating environments. A program to be run is specified as a series of tokens, the first of which is the program to run, the rest are the command line arguments. Environment variables may also be specified, as well as bindings for stdin, stdout, and stderr. A variation on the theme of binding stdout and stderr is to leave them coming to the console but to tag each line with a text prefix denoting the source. Finally, a timeout may be specified to allow for shutting down a program which gets hung (in this case, the program fails).

program ::= token+ env* stdin? stdout? stderr? stdouttag? stderrtag? timeout?

token   ::= <string>

env     ::= param

stdin   ::= filename  
stdout  ::= filename  
stderr  ::= filename

stdouttag ::= <string>  
stderrtag ::= <string>

filename ::= <string>
timeout  ::= integer

In this way a program may be defined with whatever inputs and outputs it needs, environment variables, and command line tokens. Then programs may be grouped together to implement a test, and then the tests may be run with variations in parameters.

Example¶

<interoptest>
    <run test="java-java"/>
    <run test="csharp-csharp"/>
    <run test="java-csharp"/>
    <run test="csharp-java"/>

    <tests>
        <test name="java-java">
            <support>
                <prog name="java_MainPerfListener"/>
            </support>
            <jig>
                <prog name="java_MainPerfClient"/>
            </jig>
        </test>

        <test name="csharp-csharp">
            <support>
                <prog name="csharp_MainPerfListener"/>
            </support>
            <jig>
                <prog name="csharp_MainPerfClient"/>
            </jig>
        </test>

        <test name="java-csharp">
            <support>
                <prog name="java_MainPerfListener"/>
            </support>
            <jig>
                <prog name="csharp_MainPerfClient"/>
            </jig>
        </test>

        <test name="csharp-java">
            <support>
                <prog name="csharp_MainPerfListener"/>
            </support>
            <jig>
                <prog name="java_MainPerfClient"/>
            </jig>
        </test>
    </tests>

    <programs>
        <program name="java_MainPerfListener">
            <stdouttag>SOUT</stdouttag>
            <stderrtag>SERR</stderrtag>
            <tokens>
                <token>java</token>
                <token>-cp</token>
                <token>../etch/bin</token>
                <token>etch.examples.perf.MainPerfListener</token>
            </tokens>
        </program>

        <program name="java_MainPerfClient">
            <stdouttag>COUT</stdouttag>
            <stderrtag>CERR</stderrtag>
            <tokens>
                <token>java</token>
                <token>-cp</token>
                <token>../etch/bin</token>
                <token>etch.examples.perf.MainPerfClient</token>
            </tokens>
        </program>

        <program name="csharp_MainPerfListener">
            <stdouttag>SOUT</stdouttag>
            <stderrtag>SERR</stderrtag>
            <tokens>
                <token>../etch/examples/perf/src/main/csharp/PerfListenerProj/bin/Debug/PerfListener.exe</token>
            </tokens>
        </program>

        <program name="csharp_MainPerfClient">
            <stdouttag>SOUT</stdouttag>
            <stderrtag>SERR</stderrtag>
            <tokens>
                <token>../etch/examples/perf/src/main/csharp/PerfClientProj/bin/Debug/PerfClient.exe</token>
            </tokens>
        </program>
    </programs>
</interoptest>