by JayJay Zheng
A post-development performance tuning practice for addressing Oracle WebCenter Spaces performance issues
|Write for OTN|
|Earn money and promote your technical skills by writing a technical article for Oracle Technology Network.|
Read Part 1: Configurations: Oracle WebCenter Spaces and Web Cache Server
Read Part 2: JVM Monitoring and Tuning and Oracle WebCenter Spaces Application Tuning
This article is the second in a two-part series. Part 1 focuses on Oracle WebCenter Spaces OHS Configurations and Web Cache Server Configurations. Part 2 focuses on JVM Monitoring and Tuning and Oracle WebCenter Spaces Application Tuning.
Oracle JRockit is the industry-leading Java Virtual Machine (JVM). It provides high performance to ensure reliability, scalability, manageability, and flexibility for Java applications. Oracle JRockit is well suited for running Oracle WebLogic Server.
Oracle JRockit ships with Oracle JRockit Mission Control, a set of tools for monitoring, managing and profiling Java applications running on Oracle JRockit. The Oracle JRockit Mission Control tools can retrieve a profile that is a fair approximation of the actual runtime behavior of an application with very little overhead.
Oracle JRockit was the first JVM to adopt an adaptive memory management system whose behavior is based on runtime feedback. The foremost advantage of automatic memory management is its contributions to the speed of the software development cycle. An additional advantage is that an adaptive memory manager can pick the appropriate garbage collection (GC) strategy for an application based on its current behavior, appropriately changing the number of garbage collection threads or fine turning other aspects of garbage collection strategies as needed.
All techniques for automatic memory management boil down to keeping track of which objects are being used or referenced. Objects that are no longer being used are subject to garbage collection. There are a couple of ways to perform garbage collection, such as Mark and Sweep, and Stop and Copy. The garbage collection algorithm in Oracle JRockit is based on the tri-color mark and sweep, a custom algorithm based on the traditional mark and sweep.
Garbage collection in Oracle JRockit can work with or without generations depending on the optimization objectives.
Oracle JRockit garbage collection can be adjusted to optimize for throughput or low latency.
If pause times up to several seconds are permissible, throughput can be maximized to a degree that is not possible if low latency is critical. There are many variants of high throughput GC. The simplest approach is to stop the world and then garbage collect the heap with as many threads as possible -- at least as many as the number of cores on the platform, each handling a separate part of the heap. Naturally, some synchronization between GC threads is still required for references that point to other heap parts. JRockit refers to this as parallel garbage collection.
Optimizing for low latencies is basically a matter of avoiding stopping the world, and therefore also Java execution, by any means necessary. But naturally, if the garbage collector gets too little total CPU time and it can't keep up with the allocation frequency, the heap will fill up and an OutOfMemoryError will be thrown by the JVM. We need a GC that performs large parts of its work when the program is running. JRockit refers to this kind of GC as concurrent garbage collection.
Oracle JRockit JVM configuration settings can be modified to cater to Java application performance. When Oracle WebCenter is installed for production deployment, Oracle WebLogic Server is set to production mode. However, if Oracle WebCenter is installed for development and then switched to production mode for better performance, you need to include the following parameter in the startup command:
JVM heap size directly influences garbage collection behavior. Increasing the JVM heap size permits more objects to be created before an allocation failure occurs and triggers garbage collection. This naturally enables the application to run for a longer period of time between garbage collection cycles. The heap size for Oracle WebCenter Spaces in this exercise was allocated to 2GB. For a typical Oracle WebCenter Spaces/ Oracle WebCenter Portal application, this is a relatively low heap size. It is recommended to go with the maximum memory allocation the system allows. The exception would be that several managed servers have been set up on the same physical machine and you can distribute the available memory allocation equally to the managed servers. For instance, consider a scenario in which the physical memory is 32GB and, apart from an admin server and other system resources, the maximum memory available for allocation is 25GB. If there are four managed servers set up for the Oracle WebCenter Spaces application, each one should be assigned 5GB, considering each JVM itself would consume 0.5-1GB for its engine and classes memory.
In this practice, we have four managed servers (Spaces, Collaboration, Utilities, and Portlet) and one admin server running on the system. The Collaboration, Utilities, and Portlet managed servers and the admin server will take up as much as 2.5GB each, leaving a maximum allocation of 4-5 GB for the Spaces managed server. An attempt to set the Spaces managed server to a heap size of 6GB resulted in problems with the Spaces application in this exercise. Therefore 4GB is the maximum heap size for the following practice.
For monitoring, the Oracle JRockit Mission Control client is installed on the local desktop workstation. To enable Oracle JRockit Mission Control remote monitoring, please use the following java arguments to turn on the monitor port:
The above Java arguments can be added via JVM startup commands in the Oracle Weblogic console or included in the setDomainEnv.sh file. Please note that the JVM has to be restarted for the Java arguments to take effect. The arguments can also be added manually to the Linux command to start the JVM.
To connect the JVM from Oracle JRockit Mission Control, refer to Figure 1.
Since response time is our priority, different sets of heap size and garbage collection configurations have been tested and monitored.
A downside to increased heap size is a corresponding increase in the amount of time needed to find and process objects that should be garbage collected. Therefore JVM heap size tuning often involves a balance between the interval of GC and the pause time needed to perform the GC. It is recommended to set the initial heap size and max heap size equally to maximize run time performance. This prevents the JVM from compacting, which dynamically resizes the heap, which is a very expensive operation. For the following three trials, the same initial and max heap size is used. Since the trials are not based on a defined load test, the load for each trial is different depending on the number of users and how heavily each user navigates the sites during the tests. So the number of garbage collections may not be a good indicator of the relationship between the load and how garbage collection algorithms respond. But the GC time parameter is a fairly good performance indicator for garbage collection algorithms since the GC time parameter is relys lessing on the load and more on the garbage collection strategy.
Before we start, here is the information on the different GC algorithms used during the trials:
-XgcPrio:throughput: The default setting. This optimizes for throughput, not caring about pause times.
-XgcPrio:pausetime: This optimizes for low latency.
-XgcPrio:deterministic: This activates the Oracle JRockit Real Time functionality, striving for extremely short pauses at the cost of some additional runtime overhead.
The maximum pause time target for the GC (not applicable for -XgcPrio:throughput) can be set with the flag -XpauseTarget.
JVM startup arguments:
-Xms:4096m -Xmx:4096m -Xns:2048m -XgcPrio:deterministic -XpauseTarget:100
JVM startup arguments:
-Xms:4096m -Xmx:4096m -Xns:1024m -XgcPrio:deterministic -XpauseTarget:100
JVM startup arguments:
|Trial 1||Trial 2||Trial 3|
|GC Algorithms||deterministic, Concurrent mark & sweep||deterministic, Concurrent mark & sweep||throughput, generational parallel mark & sweep|
|Heap Size||4 GB||4 GB||4 GB|
|Nursery Size||2 GB||1 GB||Dynamic|
|Pause time target||100 ms||100 ms||N/A|
|Avg GC time||786 ms||723 ms||269 ms|
|Total GC time 1||2.6 s||11.6 s||4.3 s|
Since nursery size is set above 1GB for both trial 1 and trial 2, there is no young collection, but there is a corresponding increase in old collection. Collection of old objects takes longer than young collection, causing longer average GC time and total GC times. With GC set to deterministic, 11-12 seconds are spent on garbage collection in a one hour timeframe during our test. Clearly, Oracle JRockit cannot maintain the 100ms pause time target with the current heap and system configuration. On the contrary, performance is much better with default settings with a 4GB heap size, resulting in less average GC time and much less total time. The important factor for the default setting is that the nursery size is dynamic and ranges from 0% to 95% of the initial heap size. Nursery size refers to the storage for the short-lived objects , which are collected faster than old objects. The collections between young and old objects are balanced (9 and 10), which indicates that the default setting on the JVM garbage collection is suitable for our Spaces application, considering that 4GB is the maximum heap size.
Heap usage and CPU usage indicate how the system resources respond to the page load. They are indicators of whether the current hardware resources and physical memory are suitable for the Spaces application with the testing load. When the minimum and maximum of the heap size is set at the same level (4GB in this case), the JVM will try to use the full capacity of the memory being allocated. From our testing (as depicted in Figure 5), we can see the heap size maintains at a level of above 50% of the total heap size in a one hour timeframe. After the time stamp of 9:10am, there were more frequent garbage collections going forward. The memory usage at the end of the garbage collection reflects the application memory consumption. Therefore we can see that memory climbs from 512MB to 1GB and finalizes at 1.5GB. Considering there are only a few users in the testing environment, it's most probable for the system to reach memory saturation with more users entering the application sites. Once the release of memory from garbage collection is slower than the demands from the increasing load, it is expected to see that heap usage maintains at a high level (90-100%), resulting in very long response times for users on the browser side. Therefore the heap size could be the performance bottleneck for the Spaces application in our test case. CPU usage is very low throughout the testing, and peak usage is around 35%. It's common to see CPU usage around 30-50% in average and 70-80% in peak. Therefore there is lots of potential for both CPU and memory to take more load.
The testing results and behavior should not carried over to other instances. This is because slight changes to the system environment could produce very different JVM behaviors. The practice illustrated here is recommended for other instances as well to determine the heap size allocation and garbage collection strategy for best performance tuning. Experimenting with different pause time targets for a particular application is also encouraged.
There is no doubt that performance is affected by how an application is coded and configured. In this section we'll look at overall performance metrics, and then turn our attention to an examination of Spaces application source code and and application configurations.
Oracle WebCenter Fusion Middleware Control provides powerful tools to monitor Spaces application performance in terms of Spaces services, individual spaces, pages inside a space, and other factors. Let's take a closer look at service performance.
Two types of measurements are available: Since Startup and Recent History. Since Startup refers to the data are collected since the last managed server startup. Recent History provides data on collection that has occurred in the last 10-15 minutes.
Figure 7 shows a snapshot of services metrics for an Oracle WebCenter Space. Even though all services are up and running, most of the services are not consumed by the Spaces applications. Content Repository is the exception. It could be easily identified by the Invocations attribute. The average time to invoke the content repository service is satisfactory in this case.
Within the actual Services Summary, the Content Repositories link (the second item listed under Service Name in Figure 7) points to a detailed performance metrics report in which individual operations of content service are revealed and laid out with several parameters, such as popularity of operation, response time and upload/download throughput per second. Long response time for download/upload has not been found via this practice. But the response time could be as long as several seconds depending on the actual size of the file downloaded/uploaded from/to the content server.
More detailed information on each operation can be investigated by reviewing the operation metrics in the table format, as shown in Figure 8. These metrics demonstrate the popularity of the operation by the number of invocations, and by the response time in both Since Startup and Recent History. The maximum response time for each operation is also displayed in the table. While there are a couple of operations with very high response time values, this does not necessarily indicate there are performance issues with the operations, given that the average response time is rather low (Figure 8). But this does indicate that under certain circumstances the response times of the operations are much longer than expected. This could be a result of the JVM hanging, system shutdown/restart by human operations, or other causes.
Periodic examination of performance metrics is recommended to ensure that all spaces and their pages are working properly. Aside from what is explained above, Oracle Fusion Middleware Control provides a more flexible way to monitor a particular area or functionality by customizing the metric palette. Please refer to the oracle documentation for more information on using the metric palette: http://docs.oracle.com/cd/E14571_01/core.1111/e10105/monitor.htm#HHAGICCJ
The Spaces application source code is examined from a performance perspective to achieve quicker page loading. Examination of the following areas should be focused and considered as high priorities:
It is always good practice to revisit custom code after development. Improvements can be gained by eliminating unused code (code that is not in effect for any functionality), by eliminating unnecessary comments, and by centralizing resources, and refining codes to reduce page, bypass performance hits, and so on. The following is a check list of guidelines for examining the code with the aim of gaining performance improvement:
The above items are not targeted to change the implementation of any functionality. But if slow processing is observed for any functionality, it is recommended that you refine the functionality using a different implementation, or analyze if other alternatives exist.
The page template is used on every single page; therefore any code refinement on the page template could benefit every page. After the page template is refined, it is recommended that you compare the response time for each individual page and draw an average line, starting with the page that takes the longest loading time. Custom task flow is also an area that should be investigated. A custom task flow may contain a model layer, a UI layer, or both. For a UI layer, the same rules defined above apply, since the page template, individual pages or fragments, and managed beans are involved in the UI layer. If model layer performance tuning is required — e.g. View Object Tuning, Application Module tuning Passivation/Statement Management) — please refer to the Oracle documentation for further directions. The efforts listed in this section can be repeated until a satisfactory loading time is achieved.
There are a number of factors that can impact Oracle WebCenter Spaces / Oracle WebCenter Portal performance inside application configurations. The configuration files that are examined in this practice are as follows:
|WebCenter Application||HTTP Session Timeout||web.xml|
|JSP Page Timeout||web.xml|
|ADF Client State Token||[web.xml|
|MDS Cache Size and Purge Rate||Cadf-config.xml|
|CRUD APIs (Create, Read, Update and Delete) ad||f-config.xml|
|Announcements Service||Connection Timeout||connections.xml|
|Discussions Service||Connection Timeout||connections.xml|
|Instant Messaging and Presence (IMP) Service||Connection Timeout||connections.xml|
|Mail Service||Connection Timeout||connections.xml|
|RSS News Feed Service||Refresh Interval||adf-config.xml|
|RSS News Feed Service||Refresh Interval||adf-config.xml|
|Search Service||Number of Saved Searches Displayed, Number of Results Displayed, various Timeouts||adf-config.xml|
|RWSRP Producers||RConnection Timeout||connections.xml|
|Oracle PDK-Java Producers||RConnection Timeout||connections.xml|
|Portlet Service||Locale Support, Portlet Timeout, Portlet Cache Size||adf-config.xml|
In addition to the parameters listed above, many other parameters are related to performance since they enable/disable extra features for troubleshooting/debugging, and induce more overhead on the server. Some of these features should only be enabled in development mode for specific reasons. The following is a list of parameter-value pairs that must be examined in the web.xml file. The values indicated for each parameter are considered to improve performance.
At startup, the newer application is automatically deployed with the modified web.xml.
No modification as made to connections.xml since no performance bottleneck was observed in the Spaces backend components and services. Please refer to the 'Overall Performance Metrics' section for details.
Caching rules can also be defined in the adf-config.xml file to implement caching in the Java Object Cache. Please refer to Oracle Fusion Middleware Web User Interface Developer's Guide for Oracle Application Development Framework on how to set it up.
Performance tuning is a broad topic. This article focuses on three areas that are key to improving performance (page loading time): caching and compression, the JVM, and the Spaces application. An approximate improvement of 0.3-0.5 seconds has been observed as a result of tweaking the Spaces application. With further examination of the individual Spaces pages and custom task flows, additional improvements can be expected on the individual pages that are taking longer than average. The benefits on tuning the JVM were not noticed at this stage because the current allocated heap size is still relatively low for the Spaces application. As recommended in the JVM tuning section of this article, it will be beneficial to add more memory allocation to reduce page loading time and increase response time stability.
Performance tuning is not one time job. After rounds of technical and functional testing, modifications can be reliably migrated to a different environment, such as a production instance. However, doing so will change that environment would change. For example, if there are more loads for a certain Space, the addition of a dedicated manager server could be an option. If the usage pattern for a Space is changed, resulting in changes to in the popularity of certain requests, a re-analysis of popular requests and adjustment of caching rules is encouraged. If free memory is available to allocate to a managed server, increasing the heap size and monitoring the JVM is recommend as a follow-up. Ongoing performance monitoring is recommended when there are changes in the environment, such as the introduction of new services or the publication of a new Space to a large number of users.
Performance is an important consideration, but so, too, is the scalability of the Space application. Performance can be dramatically degraded when the Spaces application reaches its maximum tolerance. It is crucial to understand how many concurrent users the Space application can accommodate before comprising response time. Therefore, it is beneficial to execute capacity planning for a clear understanding of business needs. Finally, stress testing is recommended to validate if current system resources can fulfill the expected business needs, and to determine if adjustments to system resources are necessary.
JayJay Zheng is a Solution Architect and certified Oracle WebCenter implementation specialist with AurionPro SENA. An Oracle Fusion Middleware evangelist, JayJay has over seven years of experience consulting in Oracle ADF, Oracle WebCenter Portal/Content, and SOA technologies.