SPECjAppServer2004: The New Way to Evaluate J2EE Performance

by Samuel Kounev

09/05/2005

Abstract

This article presents SPECjAppServer2004—the new industry-standard benchmark for measuring the performance and scalability of J2EE hardware and software platforms. SPECjAppServer2004 is a completely new benchmark and not comparable to the SPEC J2EE benchmarks released in late 2002. This article discusses the business domains and workload modeled by the benchmark, as well as the benchmark design and architecture. The author also explains the meaning of the benchmark metrics, discusses the different purposes the benchmark can be used, and provides some links to additional information.

Introduction

SPECjAppServer2004 is a new industry-standard benchmark for measuring the performance and scalability of Java 2 Enterprise Edition (J2EE) technology-based application servers. SPECjAppServer2004 was developed by SPEC's Java subcommittee, which includes BEA, Borland, Darmstadt University of Technology, Hewlett-Packard, IBM, Intel, Oracle, Pramati, Sun Microsystems, and Sybase. It is important to note that although some parts of SPECjAppServer2004 look similar to SPECjAppServer2002, SPECjAppServer2004 is much more complex and substantially different from previous versions of SPECjAppServer. It implements a new enhanced workload that exercises all major services of the J2EE platform, including:

  • The Web container, including servlets and JavaServer Pages
  • The EJB container
  • EJB 2.0 container-managed persistence
  • JMS and message-driven beans
  • Transaction management
  • Database connectivity

Moreover, SPECjAppServer2004 heavily exercises all parts of the underlying infrastructure that make up the application environment, including hardware, JVM software, database software, JDBC drivers, and the system network. Server vendors can use SPECjAppServer2004 to measure, optimize, and showcase their products' performance and scalability. Their customers can use it to gain a better understanding of the tuning and optimization issues surrounding the development of modern J2EE applications.

SPECjAppServer2004 Business Model

The SPECjAppServer2004 workload is based on a distributed application claimed to be large enough and complex enough to represent a real-world e-business system. The benchmark designers have chosen manufacturing, supply chain management, and order/inventory as the "storyline" of the business problem to be modeled. This is an industrial-strength distributed problem that is heavyweight, mission-critical, and requires the use of a powerful and scalable infrastructure. Most important, it requires the use of middleware services, including object persistence, caching, distributed transactions, clustering, load-balancing, resource pooling, asynchronous messaging, and dynamic Web page generation, among others. It is those services of application servers that are stressed and measured by the SPECjAppServer2004 benchmark.

The SPECjAppServer2004 workload has been specifically modeled after an automobile manufacturer whose main customers are automobile dealers. Dealers use a Web-based user interface to browse the automobile catalogue, purchase automobiles, sell automobiles, and track dealership inventory.

As Figure 1 depicts, SPECjAppServer2004's business model comprises five domains:

  • Customer domain dealing with customer orders and interactions.
  • Dealer domain offering a Web-based interface to the services in the customer domain.
  • Manufacturing domain performing "just-in-time" manufacturing operations.
  • Supplier domain handling interactions with external suppliers.
  • Corporate domain managing all customer, product, and supplier information.

Figure 1. SPECjAppServer2004 business model

Figure 2 shows some examples of typical transactions run in these domains (the dealer domain is omitted, since it does not provide any new services on itself).

Figure 2. SPECjAppServer2004 business domains

We include a brief overview of the four domains as they are described in the benchmark documentation:

The customer domain

The customer domain hosts an order entry application that provides some typical online ordering functionality. The latter includes placing new orders, retrieving the status of a particular order or all orders of a given customer, canceling orders, and so on. When orders arrive from customers, a credit check is performed on the customer by sending a request to the corporate domain. In addition, the proper discount is calculated. Orders for more than 100 automobiles are called large orders.

The dealer domain

The dealer domain hosts a Web application (called dealer application) that provides a Web-based interface to the services in the customer domain. It allows customers, in our case automobile dealers, to keep track of their accounts, keep track of dealership inventory, manage a shopping cart, and purchase and sell automobiles.

The manufacturing domain

The manufacturing domain hosts a manufacturing application that models the activity of production lines in an automobile manufacturing plant. There are two types of production lines, namely planned lines and large order lines. Planned lines run on schedule and produce a predefined number of automobiles. Large order lines run only when a large order is received in the customer domain. The unit of work in the manufacturing domain is a work order. Each work order is for a specific number of automobiles of a certain model. When a work order is created, the bill of materials for the corresponding type of automobile is retrieved and the required parts are taken out of inventory. As automobiles move through the assembly line, the work order status is updated to reflect progress. Once the work order is complete, it is marked as completed and inventory is updated. When inventory of parts gets depleted, suppliers need to be located and purchase orders (POs) need to be sent out. This is done by contacting the supplier domain, which is responsible for interactions with external suppliers.

The supplier domain

The supplier domain is responsible for interactions with external suppliers. It decides which supplier to choose based on the parts that need to be ordered, the time in which they are required, and the price quoted by suppliers. The company sends a purchase order (PO) to the selected supplier. The purchase order will include the quantity of various parts being purchased, the site it must be delivered to, and the date by which delivery must happen. When parts are received from the supplier, the supplier domain sends a message to the manufacturing domain to update inventory.

The corporate domain

This domain manages the global list of customers, parts, and suppliers. Credit information, including credit limits, about all customers is kept solely in a database in the corporate domain. This is to provide maximal security and privacy.

SPECjAppServer2004 Application Design and Workload

All the activities and processes in the five domains described above are implemented using J2EE components (Enterprise JavaBeans, servlets, and JavaServer Pages) assembled into a single J2EE application that is deployed in an application server running on the System Under Test (SUT). The only exception is for the interactions with suppliers that are implemented using a separate Web application called Supplier Emulator. The latter is deployed in a Java-enabled Web server on a dedicated machine. The supplier emulator provides the supplier domain with a way to emulate the process of sending and receiving purchase orders to/from suppliers.

The workload generator is implemented using a multithreaded Java application called SPECjAppServer Driver. The latter is designed to run on multiple client machines using an arbitrary number of Java virtual machines to ensure that it has no inherent scalability limitations. The driver is made of two components: manufacturing driver and DealerEntry driver. The manufacturing driver drives the production lines (planned lines and large order lines) in the manufacturing domain and exercises the manufacturing application. It communicates with the SUT through the RMI (Remote Method Invocation) interface. The DealerEntry driver emulates automobile dealers that use the dealer application in the dealer domain to access the services of the order entry application in the customer domain. It communicates with the SUT through HTTP and exercises the dealer and order entry applications using three operations referred to as business transactions:

  • Purchase: Places orders for new vehicles.
  • Manage: Manages the customer inventory (sells vehicles and/or cancels open orders).
  • Browse: Browses through the vehicle catalog.

Each business transaction emulates a specific type of client session comprising multiple roundtrips to the server. For example, the browse transaction navigates to the vehicle catalog Web page and then pages a total of 13 times—10 forward and 3 backward. Business transactions are selected by the driver based on the mix shown in Table 1. The actual mix achieved in the benchmark must be within five percent of the targeted mix for each type of business transaction. For example, the browse transactions can vary between 47.5 percent to 52.5 percent of the total mix. The driver checks and reports on whether the mix requirement was met.

Table 1. Business transaction mix requirements
Business Transaction Type Percent Mix
Purchase 25%
Manage 25%
Browse 50%

A relational database management system (DBMS) is used for data persistence, and all data access operations use entity beans that are mapped to tables in the SPECjAppServer database. Data access components follow the guidelines in the VLDB'02 paper Improving Data Access of J2EE Applications by Exploiting Asynchronous Messaging and Caching Services (PDF) to provide maximum scalability and performance.

Communication across domains in SPECjAppServer2004 is implemented using asynchronous messaging exploiting the Java Message Service (JMS) and message-driven beans (MDBs). In particular, the placement and fulfillment of large orders (LOs), requiring communication between the customer domain and the manufacturing domain, is implemented asynchronously. Another example is the placement and delivery of supplier purchase orders, which requires communication between the manufacturing domain and the supplier domain. The latter is implemented according to the proposal in the VLDB'02 paper Improving Data Access of J2EE Applications by Exploiting Asynchronous Messaging and Caching Services (PDF) to address performance and reliability issues.

The throughput of the benchmark is driven by the activity of the dealer and manufacturing applications. The throughput of both applications is directly related to the chosen Transaction Injection Rate, which determines the number of business transactions generated by the DealerEntry driver, and the number of work orders scheduled by the manufacturing driver per unit of time. The summarized performance metric provided after running the benchmark is called JOPS, and it denotes the average number of successful JAppServer Operations Per Second completed during the measurement interval. Readers interested in more detail on the SPECjAppServer2004 EJBs, the database model, and transactions implemented can refer to the Benchmark Design Document.

To ensure that SPECjAppServer2004 results are correctly obtained and all requirements are met, the driver makes explicit audit checks by calling certain components on the SUT both at the beginning and at the end of the run. The driver includes audit results with the run results. For a list of the individual auditing activities the driver performs, refer to the Benchmark Run and Reporting Rules.

Standard vs. distributed workload

To satisfy the requirements of a wide variety of customers, the SPECjAppServer2004 benchmark can be run in Standard or Distributed mode. The SUT consists of one or more nodes, with the number freely chosen by the implementer. The entire number of databases and J2EE servers can be mapped to nodes as required. The implementation must not, however, take special advantage of the co-location of databases and J2EE servers, other than the inherent elimination of WAN/LAN traffic.

In the Standard version of the workload, all the four domains are allowed to be combined. This means that the benchmark implementer can choose to run a single deployment unit that accesses a single database that contains the tables of all the domains. However, the benchmark implementer is free to separate the domains into their deployment units with one or more database instances.

The Distributed version of the workload is intended to measure application performance where the worldwide enterprise that SPECjAppServer2004 models performs business transactions across business domains employing heterogeneous resource managers. In this model, the workload requires a separate deployment unit and a separate DBMS instance for each domain. XA-compliant, recoverable two-phase commits (see The Open Group XA Specification) are required for business transactions that span multiple domains. The configuration for the two-phase commit is required to be done in a way that would support heterogeneous systems. Even though implementations are likely to use the same resource manager for all the domains, the J2EE servers and resource managers cannot take advantage of the knowledge of homogeneous resource managers to optimize the two-phase commits.

Conclusion

SPECjAppServer2004 provides a realistic workload for measuring the performance and scalability of J2EE hardware and software platforms. You can use it for the following purposes:

  • To compare competing J2EE hardware and software platforms in terms of their performance and scalability.
  • To study the effect of different configuration options and tuning parameters offered by J2EE platforms on the overall system performance.
  • To stress test a J2EE platform and analyze it for potential bottlenecks.
  • As a sample (blueprint) application, demonstrating J2EE best practices and design patterns for building scalable applications.

Additional Reading

Samuel Kounev serves as the release manager of SPEC's Java subcommittee and was actively involved in the development and specification of the SPECjAppServer and SPECjms set of industry-standard benchmarks. He is a BEA Technical Director and holds a Ph.D. in computer science from Technische Universitaet Darmstadt and a M.Sc. degree from the University of Sofia.