Unit test history generator
Project at SourceForge Downloads Screenshots UnitTH test history

UnitTH

UnitTH is a simple, easy to use Open Source software application for generating historical test reports from JUnit test reports. All that is needed is a set of JUnit XML reports stored in a set of folders.

Features

  • Historical summary
  • Trends
  • Graphs
  • Links to all included reports
  • Sortable statistics and tables
  • Test case pass rate trends

Use cases

Unstable test suites

Test suites that are unpredictable due to for example various timing or racing conditions might be a real pain to work with. The test suites might take some time to execute and finding which test cases that are more sporadically failing than others might be rather impossible.

UnitTH screenshot.

Main page of the UnitTH test history report.

One way to deal with this problem is to execute the same test suite repeatedly and then generate some statistics out of the reports, statistics that easily can point out which runs and which test cases that failed now and then. UnitTH is designed for cases like this, repeatedly generate a number of reports and then have them parsed by UnitTH which will generate a set of HTML pages representing the test run history.

UnitTH screenshot.

Test class runs and test case verdict spread.

Continuous integration projects

Continuous integration projects are very suitable candidates for a test history generation tool. Usually CI projects flags red as soon as a single test fails which is the ideal solution, though a situation that is not always true for all setups. The more common scenario is that over a set of runs the easiest way to determine that the code has been broken is to have a look at the test run history.

UnitTH provides useful features for this where a single test case can be checked. IS it always failing or is it sporadically failing over a set of runs with the same code base.

If the CI setup runs without a "fancy" tool that drives the runs, UnitTH can very easily be integrated to generate a test history from all test runs.

SoapUI test runs

The excellent SOA test tool SoapUI has a feature that it can generate JUnit-style reports from test suite runs made through the TestRunner components. Multiple of these runs can be parsed by UnitTH into a test history report.

Enabling JUnit-style reports.
Enable JUnit style reports and the folder to save the results to.

Test history report example

During the development of this application UnitTH was used for generating a test history report for its own unit test cases. The report can be viewed by clicking on the link below.

UnitTH - development test report history

Using UnitTH

UnitTH only runs under Java 1.5 or 1.6.
  • Unzip the downloadable zip-file
  • Copy the unitth.properties file to $HOME or to the folder from where to execute UnitTH
  • Set the correct settings in the unitth.properties files
  • Run the jar file taking a set of JUnit test report folders as input (XML format)

Input flags

  • -h --help Shows the UnitTH help.
  • -v --version Shows the UnitTH version.

Example:

$ java -jar unzippedToFolder/unitth.jar storedAwayReports/report*

The properties set in the unitth.properties file can be overridden using system properties as exemplified below.

$ java -Dunitth.report.dir=<report to folder> -jar unzippedToFolder/unitth.jar storedAwayReports/report*


Integrating UnitTH with Ant scripts

The top three targets below shall be well known. Storing the test report and generating the test history targets might be new targets to implement. Below are examples for how these can be implemented.
  • build
  • test
  • generate test report
  • store test report
  • generate test history report

Storing the test report

Create a simple target that copies away the latest generated test report. Use the same folder for all stored away test reports to make it simpler to point out which folders to use as input for the test history generation. 

<target name="store.junit.report" description="Saves away the generated JUnit test report.">
  <tstamp>
    <format property="start.timestamp" pattern="yyyyMMdd-HHmmss" />
  </tstamp>
  <copy todir="${reportStorage}/report-${start.timestamp}">
    <fileset dir="report" />
  </copy>
</target>

Generating a test history

The generate.history target takes the input folders as a single argument. The rest of the test history generation parmeters will be picked up by the unitth.properties file in the current folder in $HOME. 

<target name="generate.history" description="Generates a test history from existing Unit-reports.">
  <java jar="${bin.dir}/unitth.jar" fork="true" failonerror="true" maxmemory="128m">
    <arg value="${reportStorage}/report-*" />
  </java>
</target>

The properties read from the unitth.properties can be overridden by defining system properties. Include the sysproperty in your ant target.

<-- Example where system properties are set -->
<target name="generate.history" description="Generates a test history from existing Unit-reports.">
  <java jar="${bin.dir}/unitth.jar" fork="true" failonerror="true" maxmemory="128m">
  
    <sysproperty key="unitth.html.report.path" value="."/>
    <sysproperty key="unitth.report.dir" value="report.out"/>
    <sysproperty key="unitth.generate.exectimegraphs" value="false"/>
    <sysproperty key="unitth.xml.report.filter" value="TEST-"/>
  
    <arg value="${reportStorage}/report-*" />
  </java>
</target>

Background

UnitTH originally surfaced as a Perl script a few years back. At that point the application was used for making sound judgments about software quality in release meetings. Software with failing tests was not accepted to be released unless the problems surfaced very, very sporadically. Due to the complexity of the system under test and the test framework used the simplest option was to run the test suites a fixed number of times. Depending on the sporadic fail rates for certain test cases the release was accepted or not. Not the best of examples but fixing the test framework and test cases was simply a too large of an effort compared to running the test suites a number of times.

As the application matured it started to be used in build test cycles that ran every second hour, just to be able to pin point at what points in time faulty code was checked in to the main branch. Source updates, building and testing was a very time consuming task and if skipped it would surface very quickly in the test history. All source code checkins was simply rolled back to the time before the last successful build test run was performed.

The application UnitTH is building on the same concept as the original history generator but has been reimplemented in Java with a bunch of additional features.