Test Categorization Techniques with TestNG
Pages: 1, 2, 3

System Tests

System tests exercise a complete software application, verifying external interfaces like Web pages, Web service end points, or GUIs, work end-to-end as designed. Because these tests exercise an entire system, they are often created towards the latter cycles of development. These tests also have the tendency to have prolonged execution times, in addition to extended set up and configuration times.

For example, the following test utilizes jWebUnit to test the login functionality of a Web site. Note that a number of assumptions are made in this test, such as the URL being available and the user "tst" actually having a valid account, not to mention a trading history. These implicit dependencies usually require a configuration step preceding a test run.

public class LoginTest  {
 private WebTester tester;
 /**
  * @testng.before-class = "true"    
  */
 protected void init() throws Exception {        
  this.tester = new WebTester();
  this.tester.getTestContext().
  setBaseUrl("http://stocktrader.com");
 }
 /**
  * @testng.test
  */
 public void verifyLogIn() {
  this.tester.beginAt("/");        
  this.tester.setFormElement("user", "tst");
  this.tester.setFormElement("psswd", "t@t");
  this.tester.submit();        
  this.tester.assertTextPresent("Logged in as tst");
 }
 /**
  * @testng.test dependsOnMethods = "verifyLogIn"
  */
 public void verifyAccountInfo() {
  this.tester.clickLinkWithText("History", 0);        
  this.tester.assertTextPresent("$89.00, sold");
 }
}

Developers should run these tests locally on an as-needed basis and in a continuous integration environment. Executing these tests nightly (if they can be pulled off in an automated fashion) is a good strategy. Running them more frequently can tax system resources, especially in larger environments. With proper hardware planning and sophisticated automation, however, teams can run these tests more frequently.

Implementing TestNG Categorization

Categorizing TestNG tests into these three categories is as easy as using the group annotation described above. Having a test class with methods spanning various test granularities is unusual, therefore tagging can be effectively applied at the class level.

For example, the following class has been tagged as belonging to the unit test group. Note how HierarchyBuilderTest verifies the behavior of the Hierarchy class by depending on the HierarchyBuilder type. As this relationship ends at HierarchyBuilder, which doesn't depend on a file system or database, you can effectively consider this a unit test:

import org.testng.Assert;

/**
 * @testng.test groups="unit"
 */
public class HierarchyBuilderTest {
 private Hierarchy hier;
 /**
  *  
                        
@testng.before-class = "true" groups="unit"
  */
 private void init() throws Exception{
  this.hier = 
    HierarchyBuilder.buildHierarchy(Vector.class);
 }
 
 public void validateIsntNull() throws Exception{        
  Assert.assertNotNull(this.hier, 
    "should be something!");
 }
 /**
  * @testng.test dependsOnMethods="validateIsntNull"     
  */
 public void validateValues() throws Exception{        
  Assert.assertEquals(
    this.hier.getHierarchyClassNames().length, 
    2, "should be 2");        
 }
}
                      

Similarly, individual methods in a system test can be tagged with a system group identification as demonstrated here:

public class LoginTest  {
 private WebTester tester;
 /**
  *  
                        
@testng.before-class = "true" groups="system"
  */
 protected void init() throws Exception {        
  this.tester = new WebTester();
  this.tester.getTestContext().
  setBaseUrl("http://acme.com:8080/ppo/");
 }
 /**
  *  
                        
@testng.test groups="system" 
  */
 public void verifyView() {
  this.tester.beginAt("/");        
  this.tester.setFormElement("isbn", "900930390");
  this.tester.submit();        
  this.tester.assertTextPresent("Book in stock");
 } 
}
                      

Running categorized tests

Before checking in code to a content management system, it's paramount to run tests locally, either through a build or through an environment such as an IDE. Running categorized tests via the TestNG Eclipse plug-in is amazingly simple. By selecting the groups option in the TestNG Create, manage, and run configurations dialog, as shown in Figure 1, the list of available groups is presented with checkboxes to facilitate selecting one or more. After the desired group or groups have been selected, hit the Run button, and watch that green bar go!

Figure 1
Figure 1. Eclipse integration of TestNG (click the image for a full-size screen shot)

Running categorized TestNG tests via a build becomes a matter of defining the appropriate Ant targets for each group. For example, to run all tests belonging to the component group, the TestNG Ant task is defined with the component group specified:

<target name="testng-component" 
   depends="testng-init">
 <mkdir dir="${testng.output.dir.comp}"/>

 <testng  
                        
groups="component"
    outputDir="${testng.output.dir.comp}"
    sourceDir="${testng.source.dir}"
    classpath="${testclassesdir};${classesdir}">
   <classfileset dir="${testng.source.dir}">                
     <include name="**/*Test.java"/>
   </classfileset>
   <classpath>
     <path refid="build.classpath"/>
   </classpath>
 </testng>

</target>
                      

Accordingly, using this strategy, at least four test targets would be created. There would be three corresponding to unit, component, and system tests, and a final target that could run all three.

That's a Wrap

TestNG facilitates test categories quite easily and this is probably one of TestNG's most exciting benefits. What's more, TestNG's group annotation also helps place tests into other categories, such as batch tests, acceptance tests, and even performance tests. In fact, it seems this feature may have had an influence on the newest version of JUnit, which is planning to support test groups as well!

The next time you blindly code a new test case, consider the long-term implications on build execution times. Build scalability starts with a test categorization strategy that is run at various frequencies, and an effective test categorization strategy starts with TestNG.

References

Andy Glover is the President of Stelligent Incorporated and is based in Reston, Virginia. He actively blogs about software quality at www.thediscoblog.com.