SPECjAppServer2004: The New Way to Evaluate J2EE Performance
Pages: 1, 2
All the activities and processes in the five domains described above are implemented using J2EE components (Enterprise JavaBeans, servlets, and JavaServer Pages) assembled into a single J2EE application that is deployed in an application server running on the System Under Test (SUT). The only exception is for the interactions with suppliers that are implemented using a separate Web application called Supplier Emulator. The latter is deployed in a Java-enabled Web server on a dedicated machine. The supplier emulator provides the supplier domain with a way to emulate the process of sending and receiving purchase orders to/from suppliers.
The workload generator is implemented using a multithreaded Java application called SPECjAppServer Driver. The latter is designed to run on multiple client machines using an arbitrary number of Java virtual machines to ensure that it has no inherent scalability limitations. The driver is made of two components: manufacturing driver and DealerEntry driver. The manufacturing driver drives the production lines (planned lines and large order lines) in the manufacturing domain and exercises the manufacturing application. It communicates with the SUT through the RMI (Remote Method Invocation) interface. The DealerEntry driver emulates automobile dealers that use the dealer application in the dealer domain to access the services of the order entry application in the customer domain. It communicates with the SUT through HTTP and exercises the dealer and order entry applications using three operations referred to as business transactions:
Each business transaction emulates a specific type of client session comprising multiple roundtrips to the server. For example, the browse transaction navigates to the vehicle catalog Web page and then pages a total of 13 times—10 forward and 3 backward. Business transactions are selected by the driver based on the mix shown in Table 1. The actual mix achieved in the benchmark must be within five percent of the targeted mix for each type of business transaction. For example, the browse transactions can vary between 47.5 percent to 52.5 percent of the total mix. The driver checks and reports on whether the mix requirement was met.
|Business Transaction Type||Percent Mix|
A relational database management system (DBMS) is used for data persistence, and all data access operations use entity beans that are mapped to tables in the SPECjAppServer database. Data access components follow the guidelines in the VLDB'02 paper Improving Data Access of J2EE Applications by Exploiting Asynchronous Messaging and Caching Services (PDF) to provide maximum scalability and performance.
Communication across domains in SPECjAppServer2004 is implemented using asynchronous messaging exploiting the Java Message Service (JMS) and message-driven beans (MDBs). In particular, the placement and fulfillment of large orders (LOs), requiring communication between the customer domain and the manufacturing domain, is implemented asynchronously. Another example is the placement and delivery of supplier purchase orders, which requires communication between the manufacturing domain and the supplier domain. The latter is implemented according to the proposal in the VLDB'02 paper Improving Data Access of J2EE Applications by Exploiting Asynchronous Messaging and Caching Services (PDF) to address performance and reliability issues.
The throughput of the benchmark is driven by the activity of the dealer and manufacturing applications. The throughput of both applications is directly related to the chosen Transaction Injection Rate, which determines the number of business transactions generated by the DealerEntry driver, and the number of work orders scheduled by the manufacturing driver per unit of time. The summarized performance metric provided after running the benchmark is called JOPS, and it denotes the average number of successful JAppServer Operations Per Second completed during the measurement interval. Readers interested in more detail on the SPECjAppServer2004 EJBs, the database model, and transactions implemented can refer to the Benchmark Design Document.
To ensure that SPECjAppServer2004 results are correctly obtained and all requirements are met, the driver makes explicit audit checks by calling certain components on the SUT both at the beginning and at the end of the run. The driver includes audit results with the run results. For a list of the individual auditing activities the driver performs, refer to the Benchmark Run and Reporting Rules.
To satisfy the requirements of a wide variety of customers, the SPECjAppServer2004 benchmark can be run in Standard or Distributed mode. The SUT consists of one or more nodes, with the number freely chosen by the implementer. The entire number of databases and J2EE servers can be mapped to nodes as required. The implementation must not, however, take special advantage of the co-location of databases and J2EE servers, other than the inherent elimination of WAN/LAN traffic.
In the Standard version of the workload, all the four domains are allowed to be combined. This means that the benchmark implementer can choose to run a single deployment unit that accesses a single database that contains the tables of all the domains. However, the benchmark implementer is free to separate the domains into their deployment units with one or more database instances.
The Distributed version of the workload is intended to measure application performance where the worldwide enterprise that SPECjAppServer2004 models performs business transactions across business domains employing heterogeneous resource managers. In this model, the workload requires a separate deployment unit and a separate DBMS instance for each domain. XA-compliant, recoverable two-phase commits (see The Open Group XA Specification) are required for business transactions that span multiple domains. The configuration for the two-phase commit is required to be done in a way that would support heterogeneous systems. Even though implementations are likely to use the same resource manager for all the domains, the J2EE servers and resource managers cannot take advantage of the knowledge of homogeneous resource managers to optimize the two-phase commits.
SPECjAppServer2004 provides a realistic workload for measuring the performance and scalability of J2EE hardware and software platforms. You can use it for the following purposes:
Samuel Kounev serves as the release manager of SPEC's Java subcommittee and was actively involved in the development and specification of the SPECjAppServer and SPECjms set of industry-standard benchmarks. He is a BEA Technical Director and holds a Ph.D. in computer science from Technische Universitaet Darmstadt and a M.Sc. degree from the University of Sofia.