Jakarta Batch TCK Reference Guide

The Jakarta Batch TCK is provided under the Eclipse Foundation Technology Compatibility Kit License - v 1.0 [https://www.eclipse.org/legal/tck.php].

This guide is for implementers of the Jakarta Batch specification, to assist in running the test suite that verifies the compatibility of their implementation.

Before reading this guide, you should familiarize yourself with the Jakarta Batch Version 2.0 specification, which can be found at https://jakarta.ee/specifications/batch/2.0/.

Other useful information and links can be found on the eclipse.org project home page for the Jakarta Batch project [https://projects.eclipse.org/projects/ee4j.batch] and also at the GitHub repository home for the specification project [https://github.com/eclipse-ee4j/batch-api].

As an overview, note in order to pass the Jakarta Batch TCK you must run against your implementation, passing 100% of both the:

The two types of tests are not encapsulated in a single execution task or command; they must be executed separately from each other.

The JDK used during test execution must be noted and listed as an important component of the certification request. In particular, the Java SE version is important to note, and this version must be used consistently throughout both the TestNG and Signature tests for a given certification request.

For the current TCK version, this can be done with either Java SE Version 8 or Version 11.

The challenge process is defined within the Challenges section within the Jakarta EE TCK Process 1.0.

Challenges will be tracked via the issues of the Jakarta Batch Specification repository.

As a shortcut through the challenge process mentioned in the Jakarta EE TCK Process 1.0 you can click here, though it is recommended that you read through the challenge process to understand it in detail.

The certification of compatibility process is defined within the Certification of Compatibility section within the Jakarta EE TCK Process 1.0.

Certifications will be tracked via the issues of the Jakarta Batch Specification repository.

As a shortcut through the cerification of compatibility process mentioned in the Jakarta EE TCK Process 1.0 you can click here, though it is recommended that you read through the certification process to understand it in detail.

Note that the Ant installation used below should use the JDK installed in 1. when the 'ant' command is executed during the test execution described below (either via the JAVA_HOME variable or other typical methods).

The Jakarta Batch TCK is distributed as a zip file, which contains the TCK artifacts (the test suite binary and source, porting package SPI binary and source, the test suite XML definitions, and signature files) in /artifacts, the TCK library dependencies in /lib and documentation in /doc. You can access the current source code from the Git repository: https://github.com/eclipse-ee4j/batch-tck.

The software can simply be extracted from the ZIP file. Once the TCK is extracted, you’ll see the following structure:

In more detail:

artifacts contains all the test artifacts pertaining to the TCK: The TCK test classes and source, the TCK SPI classes and source, the TestNG suite.xml file and the signature test files.

doc contains the documentation for the TCK: this reference guide, plus a script that helps provide an example of how to run the TCK against the 'jbatch' implementation.

lib contains the necessary prereqs for the TCK

build.xml, sigtest.build.xml Ant build files used to run TestNG, signature test portions of the TCK

batch-tck-testng.properties, batch-tck-sigtest.properties Specify properties here for each of the TestNG, signature test portions of the TCK, respectively (And the remaining text files are self-explanatory.)

In order to run the TCK, you must set one required property, batch.impl.testng.path to refer to the Jakarta Batch runtime implementation that you are running the TCK against, plus any needed dependencies.

An optional property with name jvm.options is provided to specify JVM arguments using a <jvmarg line=""/> child element of the TestNG Ant task. This property value should list the desired JVM arguments, separated by spaces.

The Ant buildfile will load properties from the file batch-tck-testng.properties, which can be a convenient place to define properties like batch.impl.testng.path and jvm.options.

This batch-tck-testng.properties file also contains a set of predefined values for test-specific sleep and wait times, which may be customized for a given implementation/environment and still result in a valid TCK execution, suitable for certification.

Example:

The basic test flow simply involves a TestNG test method using the JobOperator API to start (and possibly restart) one or more job instances of jobs defined via one of the test JSLs, making use of some number of com.ibm.jbatch.tck.artifacts Java artifacts. The JobOperator is wrapped by a thin layer which blocks waiting for the job to finish executing (more on this in the discussion of the porting package SPI later in the document).

Several tests intentionally produce failures to test relevant portions of the specification, so a normal execution may cause a number of stack traces, error messages, etc. to stdout.

One reason TestNG was chosen was the ability to use a single XML file to hold excludes from a set of compiled tests. This is an easy way to update a suite after a TCK challenge, by updating the exclude list in this single XML file without having to update and rebuild Java source.

The Jakarta Batch TCK relies on an implementation of a "porting package" SPI to function, in order to verify test execution results. The reason is that the Jakarta Batch specification API alone does not provide a convenient-enough mechanism to check results.

A default, "polling" implementation of this SPI is shipped within the TCK itself. The expectation is that the typical Jakarta Batch implementation will be content to use the TCK-provided, default implementation of the porting package SPI.

Further detail on the porting package is provided later in this document, in case you wish to provide your own, different implementation.

The JobOperatorBridge is a utility/helper class in the Jakarta Batch TCK which makes use of the following system property:

using a default value of 900000 (900 seconds).

This prevents tests from "hanging" indefinitely if something catastrophic occurs causing the job to never complete (or if the porting package SPI "waiter" is never notified for some reason).

Note that some of the tests (e.g. the chunk tests involving time-based checkpointing) will take at least 15-25 seconds to run on any hardware, so any default value less than that applied to all tests would cause failures simply due to timing (and not because of any failure in the underlying Jakarta Batch implementation).

The value of 900 seconds was chosen, then, to avoid falsely reporting an error because of timing out too soon, allowing plenty of time for a test to finish executing, even on slower hardware, and leaves some time to attach a debugger.

It does not, however, provide "fast failure" in case of a hang or runaway thread.

In any case, this timeout value can be customized (say, to increase when debugging or decrease to force a faster failure in some cases).

Some of the TCK tests sleep for a short period of time to allow an operation to complete or to force a timeout. These wait times are defaulted via properties that are also specified in batch-tck-testng.properties.

As with many typical decisions regarding timeout values, we attempt to strike a good balance between failing quickly when appropriate but allowing legitimate work to complete.

These values can be adjusted if timing issues are seen in the implementation being tested. Refer to the comments in the test source for a specific test to better understand how the time value is used for that test.

For most development/debug use cases it is recommended to refer to the source in the Jakarta Batch TCK] GitHub repository [https://github.com/eclipse-ee4j/batch-tck], and to leverage the Maven automation and artifacts there using the associated documentation.

It should be documented how to use tags/releases, etc. to match the official level tested in the TCK distribution.

Though the TestNG build.xml script has a compile target, using a src property which could be set appropriately, this is an older usage that we haven’t focused on keeping updated. More recently we have focused on Maven automation.

Note too that for an implementation to pass the TCK, it must run against the shipped TCK test suite binary as-is (and not against a modified TCK).

We do not prescribe a certain version/distribution of signature test library. In testing the TCK (in the com.ibm.jbatch.tck.dist.exec module), we use the version of sigtestdev.jar released to Maven Central under coordinates net.java.sigtest:sigtestdev:3.0-b12-v20140219 (the JAR is here), in spite of the fact that the POM comments mention that this is an "unofficial" release.

Some alternate suggestions:

It is assumed all these options will give similar results.

The Jakarta Dependency Injection and the Jakarta Contexts Dependency Injection specifications provide a couple key dependency API classes, including:

It is not obvious that the above annotations are necessary to completely define the "signature" of some Jakarta Batch classes, which is why we call out this detail here.

The TCK package contains the signature file: batch.standalone.tck.sig_2.0_se8 in the artifacts directory.

Run the signature test by executing a command like the following (from the artifacts directory):

Note the variables above, the valuesof which you may need to modify:

Here is an example showing a sample set of values for the shell variables used in the shorthand above.

It assumes:

So with the above assumptions, the directory structure would look like:

The output of your execution should include, at the very end:

Again, in order to pass the Jakarta Batch TCK you have to make sure that your API passes the signature tests.

For additional confirmation that the signature test is working correctly, a failure can be forced by removing the last classpath entry. E.g., continuing the last example, if we remove the jakarta.enterprise.cdi-api JAR and instead do:

You will see a failure like:

We also provide a sigtest.build.xml which should typically do a good job encapsulating the java execution described above. It uses the batch-sigtest-tck.properties file to supply the four classpath entries detailed above.

It also detects whether it’s running against a Java 8 vs. 11 JVM, and can switch signature files accordingly.

We list the "raw" `java -jar …​ ` approach as the "official" one but this may be helpful as a convenience, and with such a thin wrapper it should be easy enough to agree whether results should be valid.

For reference, note that the signature tests for the TCK were generated by building a similar classpath as the one used for executing the tests, but using the 'Setup' argument in place of the 'SignatureTest' argument, e.g.:

Assuming JAVA_HOME points to the Java 11 JDK you’re using to certify, you need to:

Having done the jimage extract, you can now run the signature test by executing a command like the following (from the artifacts directory):