Before we discuss the obtained results we will first give an overview of the test setup used.
For benchmarking purposes SIPp is used. SIPp is a free Open Source test tool/traffic generator for the SIP protocol. It can generate SIP traffic and establish and release multiple calls. It can also read custom XML scenario files, describing anything from very simple to complex call flows. It includes a few basic SIPstone defined test setups. All tests in this article were performed using SIPp and the previously specified scenario.
All tests were performed using a dual Opteron 242 HP DL 145 with 2GB of memory for the proxy. The clients were run on AMD athlonXP 1600+ machines with everything interconnected in a 100Mb switched Ethernet network. All platforms were running Debian GNU/Linux with a 2.6 kernel. Sun's JDK 1.4.2 was used.
We can now provide an overview of the test results. The previous sections examined the internals of the Java Virtual Machine, how the memory is managed, and what garbage collection options currently are available. Furthermore we examined how these options can be tuned to optimize the behavior of the virtual machine and the garbage collector. We specifically focused on minimizing the pauses introduced by garbage collection as they may cause the setup of a SIP call to fail. We'll now see how this all fits together by running the tests against the BEA WebLogic SIP Servlet implementation.
The SIP Servlet container was submitted to a number of test runs that allowed us to evaluate the garbage collection tuning options. When setting up a SIP call using the scenario specified in Figure 1, the acknowledgment of the INVITE message should arrive within 50ms after sending it for 95 percent of the calls and it should arrive within 25ms for 50 percent of the calls.
Figure 8 shows the performance results of the SIP Servlet application server when no special tuning parameters were set, except for the maximum heap size. It clearly shows that the 95th percentile rises very quickly starting at 50 calls per second (caps). At this point multiple SIP messages start arriving concurrently which means some messages are first queued before they can be processed.
Figure 8. SIP Servlet results with default garbage collection
Another problem with this result is the low CPU usage. At the higher call rates the CPU-usage stays relatively low but some calls do time out, which of course is not desirable. Taking into account that 95 percent of the calls should be answered within 50ms, the server would only be able to handle 130 caps.
Figure 9 shows the results of a test run with the optimized low latency options from Table 1 and the pretenuring options. The first noticeable difference is the much slower increase of the 95th percentile. Although there still is an increased delay starting at 70 caps the increase is much less steep. As the average garbage collection pauses is much lower than that with the default options, the majority of the calls will not be delayed for too long. With the default garbage collection options pauses, up to 1 second could occur as opposed to less than 10ms for the optimized options.
Figure 9. SIP Servlet results with optimized garbage collection tuning
Secondly we notice the CPU usage is higher and reaches almost 100 percent at 250 caps. Starting at this call rate some calls do start to timeout. Fortunately we do get the better response times in return. Obviously the more advanced garbage collector algorithms require more system resources, but results in a much better job. The requirement for the 95th percentile is now met up to 240 caps, which is a significant increase compared to the 130 caps reached with the default garbage collector.
For completeness initial test runs with the minimal execution time optimization options from Table 2 were performed as well. The results of these tests did show an improvement over the default garbage collection regarding the obtained latency. However, using these options results in longer garbage collection pauses, which cause a relatively large number of calls to time out. At 150 caps approximately 10 percent of the calls did time out. The calls that did get processed were answered well within the requirements, but a loss rate of 10 percent is unacceptable for this type of application. Nevertheless, these options can be suitable for other types of applications with less stringent requirements or in cases where it is less important for every single call to be answered as soon as possible.
Java technologies such as SIP Servlet are currently available for telecom and IMS platform design, simplify the management and security enforcement, and allow for a smooth integration with existing systems. This article shows that performance problems caused by the garbage collector of the Java Virtual Machine can be solved by specifying two sets of tuning options that will improve the low latency behavior of the garbage collector or will minimize the total execution time.
As VoIP applications have very strict requirements when it comes to low latency and high throughput, a multimedia call setup was chosen as a real-world test case for the validation of the proposed optimizations. The evaluation results show that Java and SIP Servlet can meet the strict requirements of low latency telecom applications if appropriate tuning for the Java Virtual Machine is used.
Bruno Van Den Bossche is a Ph.D. student affiliated with the Department of Information Technology at Ghent University. His main research interests include scalable software architectures, distributed software and the automatic optimal distribution of software.
Filip De Turck is a part-time professor and a post-doctoral fellow of the F.W.O.-V., affiliated with the Department of Information Technology of the Ghent University.
Bart Dhoedt works in the Department of Information Technology at the University of Ghent. His research interests are software engineering and mobile & wireless communications.
Piet Demeester is a professor at the Ghent University where he is responsible for the research and education on communication networks.