JavadocTest is a plugin for the javadoc API documentation generation tool that runs test code contained in Javadoc code snippets.

Test automation is part of any good Continuous Integeration (CI) phase of the software development lifecycle. Unit tests are written to check the output of functions and methods. These tests are run by developers or agents during the development process, and typically as part of the automated CI pipeline. Test code is typically written in dedicated files that call the code to be tested. Usually, project code and test code in the same source code repository, with a build process that does not include the test code in the resulting deliverable code artifacts.

JavadocTest allows developers to include tests within the generated API documentation of Java code, allowing it to be executed whenever the documentation is generated during CI runs, as well as having the test code itself appear directly in the developer documentation, where it can be shown alongside the classes and methods that are being tested.

Javadoc

Javadoc is the standard API documentation generation tool for programs written in Java. It scans the source code to find code comments in specific locations, creating a set of HTML pages that provide documentation for Java modules, packages, classes, methods, and more.

For example, a method within a class can include developer documentation as a specially formatted comment.

/**
 * Find the greatest common divisor of two integers.
 *
 * @param a The first integer.
 * @param b The second integer.
 * @return the greatest common divisor of a and b.
 */
public static int gcd(int a, int b) {
    if(b == 0) {
      return a;
    }
    return gcd(b, a % b);
}

Java 18 added "snippet" support to javadoc: this introduces syntax that tells the tool that a region of the documentation comment is sample source code. These are formatted as code in the generated HTML, and include a convenience link to copy the contents of the snippet into the web browser host's clipboard, which simplifies copying and pasting code snippets into code editors.

/**
 * Find the greatest common divisor of two integers.
 *
 * {@snippet :
 *   int a = 15;
 *   int b = 25;
 *
 *   int c = gcd(a, b);  // c is 5.
 * }
 * @param a The first integer.
 * @param b The second integer.
 * @return the greatest common divisor of a and b.
 */
public static int gcd(int a, int b) {
    if(b == 0) {
      return a;
    }
    return gcd(b, a % b);
}

Doclets and JavadocTest

The javadoc tool has a plugin model; it can be run with an argument that designates which "Doclet" to use. When javadoc is run without designating a Doclet, it uses the "Standard Doclet" which generates the familiar set of HTML pages. However, javadoc can be run with a different Doclet, which can be used to process documentation comments in nearly any way.

A Doclet does not necessarily need to generate documentation artifacts. JavadocTest is a Doclet that detects snippets that have a "test" attribute The code within each snippet is turned into a Java class and run. If the snippet code compiles successfuly, and no assertions fail or uncaught exceptions are thrown, the test is considered to pass.

/**
 * Find the greatest common divisor of two integers.
 *
 * {@snippet test:
 *   int a = 15;
 *   int b = 25;
 *
 *   int c = gcd(a, b);
 *   assert c == 5;
 * }
 * @param a The first integer.
 * @param b The second integer.
 * @return the greatest common divisor of a and b.
 */
public static int gcd(int a, int b) {
    if(b == 0) {
      return a;
    }
    return gcd(b, a % b);
}

JavadocTest accepts import statements, which allow the test to use classes from outside the package. Imports can either appear as inline comments (ie, comments that lead with two slashes), and both static and non-static imports are allowed.

/**
 * Find the greatest common divisor of two integers.
 *
 * {@snippet test:
 *   // import com.example.Pair;
 *   Pair pair = getIntPair(15, 25);
 *
 *   int a = pair.first();
 *   int b = pair.second();
 *
 *   int c = gcd(a, b);
 *   assert c == 5;
 * }
 * @param a The first integer.
 * @param b The second integer.
 * @return the greatest common divisor of a and b.
 */
public static int gcd(int a, int b) {
    if(b == 0) {
      return a;
    }
    return gcd(b, a % b);
}

Imports can also be specified in the snippet's "import" attribute, in which case, the imports will not be visible in the snippet when using the standard doclet.

/**
 * Find the greatest common divisor of two integers.
 *
 * {@snippet test import=com.example.Pair:
 *   Pair pair = getIntPair(15, 25);
 *
 *   int a = pair.first();
 *   int b = pair.second();
 *
 *   int c = gcd(a, b);
 *   assert c == 5;
 * }
 * @param a The first integer.
 * @param b The second integer.
 * @return the greatest common divisor of a and b.
 */
public static int gcd(int a, int b) {
    if(b == 0) {
      return a;
    }
    return gcd(b, a % b);
}

Multiple test snippets can be used in each Javadoc comment, and each test will be run when javadoc is run with the JavadocTest Doclet.

Documentation tests are not a replacement for traditional unit tests- they are a way to communicate to developers how the feature in test is to be used, and provides another layer of quality control to the code.

Running JavadocTest

JavadocTest can be run on the command line; the doclet jar file can be downloaded from the GitHub releases page.

javadoc \
    -quiet \
    -docletpath /path/to/javadoctest-1.0.1.jar \
    -doclet org.joev.javadoctest.JavaDocTestDoclet \
    -sourcepath /path/to/source \
    -J-enableassertions \
    -J-Djavadoctest.classpath=/path/to/classes \
    package.name

JavadocTest is published to Maven Central as org.joev:javadoctest, which allows it to be included in Maven and Gradle build files.

For Maven, the maven-javadoc-plugin can be configured to run JavadocTest during the test execution phase:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-javadoc-plugin</artifactId>
  <version>3.12.0</version>
  <executions>
    <execution>
      <phase>package</phase>
      <id>standard</id>
      <goals>
        <goal>javadoc</goal>
      </goals>
      <configuration>
        <encoding>UTF-8</encoding>
        <additionalOptions>
          <additionalOption>--syntax-highlight</additionalOption>
        </additionalOptions>
      </configuration>
    </execution>
    <execution>
      <phase>test</phase>
      <id>javadoctest</id>
      <goals>
        <goal>javadoc</goal>
      </goals>
      <configuration>
        <doclet>org.joev.javadoctest.JavaDocTestDoclet</doclet>
        <docletArtifact>
          <groupId>org.joev</groupId>
          <artifactId>javadoctest</artifactId>
          <version>1.0.1</version>
        </docletArtifact>
        <sourcepath>src/main/java</sourcepath>
        <useStandardDocletOptions>false</useStandardDocletOptions>
        <disableNoFonts>true</disableNoFonts>
        <quiet>true</quiet>
        <encoding>UTF-8</encoding>
        <additionalJOptions>
          <additionalJOption>-J-enableassertions</additionalJOption>
          <additionalJOption>-J-Djavadoctest.classpath=${project.build.outputDirectory}</additionalJOption>
        </additionalJOptions>
      </configuration>
    </execution>
  </executions>
</plugin>

Gradle can be configured by adding a dependency to JavadocTest and a task to run it.

configurations {
  javadocTest
}

dependencies {
  javadocTest "org.joev:javadoctest:1.0.1"
}

task javadocTest(type: Javadoc) {
  source = sourceSets.main.allJava
  options.docletpath = configurations.javadocTest.files.asType(List)
  options.doclet = "org.joev.javadoctest.JavaDocTestDoclet"
  options.JFlags = [
    "-enableassertions",
    "-Djavadoctest.classpath=build/classes/java/main:" +
      "${configurations.compileClasspath.asPath}"]
}

Source code for JavaocTest can be found at GitHub.

Inspiration

Javadoctest is inspired by rustdoc, a tool from the Rust programming language that is the Rust equivalent of javadoc, which can run tests embedded within documentation comments.