Oracle E-Business Suite Standard Benchmarks

Note: The Oracle E-Business Suite Benchmarking Kit is no longer being provided to partners. The information on this page is provided only for historical reference for previously published benchmarks.

1. Benchmark Structure

The Oracle R12 E-Business Standard Benchmark follows the E-Business 11i (11.5.10) benchmark model by combining Online transaction execution by simulated users with concurrent Batch processing to model a "Day-in-the-Life" scenario for a global enterprise. Initially, benchmark kits are to be offered in 'Extra-Small,' 'Small,' and 'Medium' sizes with a 'Large' kit to follow

Partners can elect to execute the standard benchmark, which includes the OLTP and batch workloads or a partial benchmark consisting of just OLTP execution (potentially for WAN testing) or just Batch execution (avoids the need for load driving systems).

2. Benchmark Execution

Partners executing this kit can obtain instructions, scripts, and a copy of an expanded Oracle database with sample data. They provide their own hardware, diagnostic and analytic tools, and load driving system. Typically, partners tune the online transaction execution and the two batch jobs separately before attempting an auditable run. Note that Oracle may be able to bundle the OATS (Oracle Application Testing Suite) load tool with the Toolkit in the future. Also note that benchmarks can only be executed on supported versions or combinations of Oracle and ‘partner’ software products and hardware products.

The run starts with the ramp-up of the simulated online transaction users. When the desired number of users is 'up and running' in a stable fashion, the Order-to-Cash batch job is started-initiating data sampling. Thirty minutes after the Order-to-Cash job starts, the Payroll batch job is started. The payroll batch job can be started any time after thirty minutes from the time when Order-to-Cash starts. Sixty minutes (one hour) after the Order-to-cash job starts, all batch execution must be complete and the data sampling period ends. The online users are ramped down. This timeline is subject to change as concurrency issues and checkpoints are evaluated. Changes would be communicated through the benchmark workgroup.

This timeline is modified appropriately for the execution of a partial benchmark.

3. Submitting Results

Partners are required to submit the recorded OLTP response times and the Batch execution times along with detailed supporting information for auditing by Oracle. The collection of logs, output of audit scripts, etc. are collected to insure transparency and reproducibility. Disclosure of tuning actions can also directly benefit Oracle's customer base.

4. R12 Vs 11.5.10

For continuity and trending, differences between the R12 and its predecessor benchmark-11.5.10 have been kept to a minimum. However, partner input and Oracle directives have led to a few noteworthy updates.

Four of the ~30 OLTP transactions have been updated from 'Forms-based' to 'Web-based.' Three of the batch processes that had been 'single-threaded only' are now multi-thread capable.

11i (11.5.10) runs were set up for 'maximum user load' (100%), '90% user load,' and '70% user load.' For R12 this is replaced with 'maximum user load' (100%), '50% user load' and 'single-user' reference times. The single user reference time is optional in benchmark submissions until OATS can collect this data.

5. Posting Results

Once a partner's results have passed audit, they will be summarized in a report to be posted on Oracle's website. The report highlights the specific results and provides supporting information about the workload, hardware implementation, system utilization, software versions, tuning steps and so forth. Proper disclosure of how the results were achieved is essential to credibility in the marketplace.

6. Performance Claims

Oracle recognizes that partners invest substantial resources in undertaking the R12 E-Business Standard Benchmark and will wish to share their accomplishments with the marketplace. Nevertheless, Oracle policy is to maintain a level playing field and to insist on civil discourse and technical transparency in performance claims. Performance claims can be made, using the following rules.

Primary Metrics for Comparison
The primary metrics for EBS R12 benchmarks are listed below. The primary metrics are the only metrics that can be published or used in fence claims. No other benchmark performance metrics are allowed for claims or comparison.

• Number of Online users
• Number of Order Lines per Hour
• Number of Checks per Hour

Minimum Data for Disclosure
The publication of a performance claim in marketing materials, or any other external publication, must include the corresponding minimum data. Anyone can reference any other published benchmark as long as the minimum data is included in the reference and the claim meets the criteria set in the fence claim section.

• EBS version, database version, and OS version
• Server configuration - Processor chips, cores, threads, frequency, cache and memory
• Primary benchmark metrics
• Model size (comparisons of results of different size models is not allowed)
• Date that the claim is validated, as an "As of" date. Note that this is not the date that the benchmark is published, but is the date that the claim was validated.

Fence Claims
The intent is to document a process for making comparisons or leadership claims with the goal of reducing incomplete or flawed comparisons. Claims must only make reference to the primary metrics. Fence claims should be segmented into the following categories:

• Number of processor chips, cores or threads
• Operating Systems
• Windows
• Linux
• Unix

Price Performance Claims
Price performance claims are not allowed because pricing information is not part of the eBS benchmark submission. Therefore, performance claims would lead to inaccurate or non-standardized price performance results.

Allowable Fence Claim Examples

• A new record was set by achieving the highest Oracle eBS R12 large model size result
  • All 3 primary metrics must show highest result in order for this claim to be made by a vendor
• Best 4-core performance on Oracle eBS R12 medium kit.
  • Model size and benchmark version is required
• Highest Oracle eBS R12 medium size result on Linux
  • This claim is valid as operating system is a fence claim
  • No other Linux result of equal user count can be published for this claim to be used. If equal result is published then vendor must use additional fence claim. i.e. processor chips, cores or threads
• Per core leadership on Oracle eBS R12 Payroll Batch medium model size
  • This claim is valid as core is a fence claim. It must show the checks per hour in the minimum data disclosed. The number of workers/threads should be documented for batch comparisons if the amount is different in the results being compared.
• Best Linux per processor/chip result on Oracle eBS R12 small model size
  • This claim is valid as operating system and processor chip are fence claims
• Highest per core Unix performance on Oracle RAC eBS R12 large model size
  • This claim is valid as RAC or single instance database is a fence claim

Not Allowed Fence Claim Examples

• Best 8-core Oracle eBS R12 medium model sizebased on overall response time
  • Response time is not the primary metric and is therefore not allowable for publication.
• Price performance leadership on Oracle eBS R12 small kit
  • Pricing data is not a metric in the OASB benchmark, and therefore is not allowable for publication
• Best overall Oracle eBS result, or Achieves new Oracle world record
  • Too broad of a statement; would require mention of kit size and version
• Performance claims that extrapolate higher user counts than audited
  • Claims can only be made on published results, no extrapolations or estimates can be published. i.e. The system ran 3000 users @ 45% utilization. This wouldn't allow a claim that the system could run 6000 users at 90%
• eBS R12 benchmark results should not be compared to 11i or previous benchmark results.
• RAC results should not be compared to single instance results.
• Comparisons of performance results of different size benchmark kits are not allowed.
  • Highest users per processor chip at 500 users on Oracle eBS r12 small model size achieves leadership against a competitor's 300 users per processor chip on Oracle eBS r12 medium model size. This would not be a valid or allowed claim due to workload differences.

7. Oracle EBS Benchmark Workgroup Meetings:

Workgroup conference calls will be arranged and hosted by Oracle, on a quarterly minimum basis. These meetings will be the focal point for Oracle to share news about benchmark and process changes with the members and for members to submit changes to the overall benchmark process and to this document. Oracle will maintain a process for the members to propose changes. Oracle may elect to implement changes or to submit the proposed change to a vote by the members. Each company that is an active member of the workgroup will have one vote.

Revision History:
* Fence Claims Rules last updated in January 2012.

APPEAL PROCESS

This is an appeal process for challenging errors in benchmark execution, documentation, benchmark test results, or errors in publishing benchmark claims. An appeal should be carried out in the following steps:

The appealing member representative should send a note explaining the issue to the Oracle chair prior to the next Workgroup Meeting. Oracle encourages the members to work out a resolution through their benchmark member reps. If this does not work, then at the next meeting there would be a status update and formal corrective actions would be requested. Oracle has the option of making a decision on the problem or submitting it to the Benchmark Partner Workgroup for a vote. Issues should be corrected within 14 days, or as agreed upon in advance by the Oracle and the members involved. If issues are not resolved, then corrective action may be necessary which could include formal request for corrective action by Oracle, temporary delisting of the benchmark from the Oracle website or, up to, temporary loss of voting status.