10 Test Scripts
The test script is responsible for running a test, recording all the details in a TestResult object, and returning the test’s status (pass, fail, error) to the JavaTest harness. The test script must understand how to interpret the test description information returned to it by the test finder. The test script breaks down the execution of the test into a series of logical steps based on information from the test description and the test execution model. The test script can run the test itself or delegate all or part of that responsibility to commands . A fresh, new copy of the test script is created for each test. This design allows you to create test scripts for different test suites that use the same commands, much like shell and batch scripts are composed from different operating system commands.
Design Decisions
One of the most significant design decisions that you make is how the test script executes tests. The mechanism that you design can be very simple but inflexible, or it can be more complex and much more flexible.
Simple Test Scripts
Simple and less flexible test scripts construct test command lines directly from the test description and the test environment.
At the most simplistic level, scripts can execute tests using Runtime.exec . For example using the JDK:
Runtime r = Runtime.getRuntime(); String[] cmd = ; String[] env = ; Process p = r.exec(cmd, env); // read output from test using // p.getInputStream() and p.getErrorStream() // (best done with a separate thread for each stream) int rc = p.waitFor(); Status s = (rc == 0 ? Status.passed("OK") : Status.failed("process exited with return code " + rc); // s contains result status from executing command In this case the test script is responsible for collecting the test’s exit status.
The JavaTest harness provides a number of library commands that the script can use to execute system commands in different execution environments; these are described in Appendix A. One example is the library command named com.sun.javatest.lib.ProcessCommand . ProcessCommand executes a system command in a separate process on the same machine running the test script. For example:
String[] args = ; PrintWriter out1 = . // create error message stream PrintWriter out2 = . // create output message stream Command cmd = new ProcessCommand(); Status s = cmd.run(args, out1, out2); // output from command will be written automatically to // the out1 and out2 streams // s contains result status from executing command
The result of the command is a Status object based upon the exit code of the process. The exit code is analyzed by the test script and factored into the final test result. For example, if a script is executing a test by means of a series of commands and one of them fails unexpectedly, the execution may stop at that point.
More Flexible Test Scripts
More sophisticated and flexible test scripts use command templates to create custom commands. Command templates are designed by you and are created by the configuration interview from configuration information and test description information (see Chapter 6). Command templates can be created with some components of the template specified in the form of variables that the test script resolves when it uses the command to run a test. A configuration interview may provide several different templates; the script chooses among them as required for each individual test.
For example, a configuration interview might create a custom command template named command.testExecute that can be used to run all of the tests in a test suite.
command.testExecute=com.sun.javatest.lib.ProcessCommand \bin\java.exe -classpath $testSuiteRootDir\classesJDKC:\ $testExecuteClass $testExecuteArgs
The test script sets the value of the variables ($testExecuteClass and $testExecuteArgs) for each test . To review the parts of the template see Example 1.
The use of variables allows you to create flexible commands that can be used with all of the tests in the test suite. The following test script fragment shows how a test script invokes the testExecute command Foot 1 whenever it runs a test. Note that the test script uses its invokeCommand() method to execute commands:
import com.sun.javatest.*; class MyScript extends Script < public Status run(String[] args, TestDescription td, TestEnvironment env) < . // Extract values from the test description String executeClass = td.getParameter("executeClass"); String executeArgs = td.getParameter("executeArgs"); . // Set variables in the template env.put("testExecuteClass", executeClass); env.put("testExecuteArgs", executeArgs); // Invoke the command Status s = invokeCommand("testExecute"); . return s; >> In this example, the test script executes a single command for each test — the test scripts can also execute complex, multi-part tests that may involve multiple command invocations. The following examples describes some common multi-part test scenarios.
Example 1
Compiler tests generally require a multi-part test script. To test the Java compiler two stages are required:
- The compiler compiles test files
- The output from that compilation is run to ensure that it executes as expected
Example 2
Distributed tests are required to start a process on a remote system with which the test interacts. This requires a multi-part test that:
- Sets up the remote system
- Runs the primary test class that interacts with the remote system
The JavaTest harness is shipped with the source to a sample test script (StdTestScript.java) that you can refer to in the jt_install \examples\javatest\sampleFiles directory.
See the Script API documentation for information about the Script class.
Writing Custom Commands
Commands are the means by which the JavaTest harness invokes platform or test components to perform a step of the test execution model embodied in a test script. The JavaTest harness provides standard commands that are suitable for most uses, including test systems that can execute programs in a separate address space, and test systems that provide a single Java virtual machine.
If none of the standard commands are suitable, you can write a new one tailored to the test suite’s specific requirements. One scenario that requires a custom command is when the test suite uses a single JVM, and the test invokes a program that does not have a standard interface that can be used by one of the standard commands. In this case, you can write a very simple converter command that connects the interface expected by the JavaTest harness with the interface provided by the program.
The class for a command is similar (apart from the name) to the standard Test interface. The full class name is com.sun.javatest.Command .
The args argument is constructed in and passed down from the script that invokes the command. Output written to the out1 stream and out2 stream is recorded in the appropriate test result file.
Example 10-1 is an example of a command that invokes a compiler in the same JVM as the JavaTest harness, using an API for the compiler. The example uses the JDK compiler which is usually invoked directly from the command line; however, in this case an undocumented API is used. The details of how to create the PrintStream outStream from the PrintWriter out are omitted here for simplicity; the main point is to illustrate how easy it can be to write a wrapper class that passes arguments through to a non-standard API, and converts the results into the format that the JavaTest harness requires.
See the source code for JavaCompileCommand in the jt_install \examples\javatest\sampleFiles directory for a complete, commented example.
Example 10-1 JavaCompileCommand
public class JavaCompileCommand implements Command < public Status run (String[] args, PrintWriter out1,PrintWriter out2) < PrintStream outStream = . // create stream from out sun.tools.javac.Main compiler = ; new sun.tools.javac.Main(outStream, "javac") boolean ok = compiler.compile(args); return (ok ? Status.passed("Compilation OK") : Status.failed("Compilation failed")); >> For information on the standard commands provided with JavaTest. see Appendix A.
Test Result
To store test results, the JavaTest harness creates and maintains a TestResult object for each test. The test script stores information in a TestResult object while it executes a test. This information is presented to users in the JavaTest GUI and is useful when troubleshooting test runs. The more information the test script provides, the easier it is for the user to understand what occurred when the test was run.
The TestResult object contains elements:
Table 10-1 Test Result Object Elements
The test description used for the test.
The portions of the environment used to run the test. This information is displayed to the user in the Configuration tab of the JavaTest GUI.
Information about the test run. For example, start time, end time, This information is displayed to the user in the Test Run Details tab of the JavaTest GUI.Note: The test script has access to this field and can write additional information using the TestResult API.
Test output messages. This section is written by the Script class’s invokeCommand() method. This section contains at least two subsections, one for messages from the test script and one for each part of the test (if it is a multi-part test). This information is displayed to the user in the Test Run Message tab of the JavaTest GUI.
When a test completes execution, the JavaTest harness writes the results to the file testname .jtr in the work directory. Test result files are created in directory hierarchies analogous to the hierarchies in which the tests are organized.
See the API documentation for the TestResult class.
Footnote 1: When the command is invoked, the » command. » prefix is not used.