|By Janice J. Heiss, September 2009|
A Java Champion since September 2005, Kirk Pepperdine is a primary contributor and consultant to javaperformancetuning.com, which is widely regarded as the premier site for information about Java performance tuning, and the coauthor of Ant Developer's Handbook . He has been heavily involved in application performance since the beginning of his programming career and has tuned applications in a variety of languages: Cray Assembler, C, Smalltalk, and, since 1996, the Java programming language. He has also worked on building middleware for distributed applications.
Pepperdine began his career in biochemical engineering and has researched high-performance liquid chromatography for Ottawa University and the National Research Council of Canada. He worked with Cray supercomputers at the Canadian Department of Defense, as a consultant at Florida Power & Light, and has been a senior consultant with GemStone Systems. He is currently an independent consultant and an advisor to theserverside.com.
JavaSunCom (JSC): In your talks, you always warn developers that they must carefully evaluate any generic advice that you provide to see if it applies to a particular situation. Why is this so important?
Pepperdine: While I do give generic advice, I carefully explain that people have to evaluate it to see if it will work. I am giving advice in a vacuum, so what may work for many people most of the time may be very detrimental in other contexts. I don't know in advance what specific problem someone is working on.
So people need to ask: Does this advice really apply to me, or will it hurt my application performance? Do I really need to pay attention to this? I like to give people general ideas about what might be going on and how they might proceed when they have certain kinds of problems.
Performance tuning is still an art where limits are stretched and one goes to places never gone to before. It's an experimental and creative realm with no one-size-fits-all answers to performance problems. It's highly localized.
JSC: What misconceptions do you encounter among developers about performance tuning?
"All developers believe they are good at performance tuning … but generally I find that most developers are not very good at performance tuning."
Pepperdine: Here's a major one: All developers believe they are good at performance tuning. They might be very good at writing performance code, and they might be very good at coding, but generally I find that most developers are not very good at performance tuning.
I have some evidence to back this up. In my performance-tuning course, I provide an exercise that looks trivial on the surface, but developers are rarely able to get through the exercise in the session — the success rate hovers around 5 percent.
If I give the same exercise to testers, about 70 percent are successful at performance tuning the code. When Jack Shirazi, who I work with, and I first encountered this, we were very puzzled and tried to figure out what was going on. It kept happening over and over.
I can now predict who will be successful and who will not be, and that relationship is inversely proportional to the skills of the developer, strangely enough.
JSC: Inversely proportional?
Pepperdine: Yes, the worst developers tend to do better on the exercise. We discovered this when we first started doing the performance-tuning course. The very good developers are also very successful, but the group in the middle going up to the level of "very good" have problems doing it.
It turns out that performance tuning is a different skill from writing performance code and requires different thought processes and techniques than most developers are used to exercising.
"Performance tuning is a different skill from writing performance code and requires different thought processes and techniques than most developers are used to exercising."
When developers are put in situations where they are asked to performance tune, they typically look at code and take some action. But they invariably forget the dynamics of the system. If you don't include the dynamics of the system when you performance tune or if you think you understand the dynamics of the system and guess wildly wrong — which is quite often the case — you end up doing the wrong thing, which frequently happens.
Testers who take the course do far better than the developers. They will be given a process and follow it, while developers will throw it away and start mucking around, and they get burned thinking they know the code. The reason that testers are so successful is that they already have those thought processes built into their mental functioning.
We label the exercise our "reset button" because it levels the playing field. People sometimes find this to be a mind-altering experience.
JSC: Are there developers who should never worry about performance tuning?
Pepperdine: Yes, there are many applications in which performance tuning doesn't matter. I recall an app where a developer analyzed web logs once a month. He launched it and went out and had a cup of coffee during the processing.
But when you include the user in the equation, then response times become very important, as poor response time breaks the flow of one's concentration. If it takes too long, you may abandon a task because waiting becomes very painful and the user may think of other things that need to be done and abandon the process.
E-commerce involves real money, so breaks are very bad. Companies know that a three-second break will lead to a particular abandonment rate and that a four-second break will cause a higher rate of abandonment. This forces them to pay very close attention to performance.
JSC: In 2006, you wrote, "I have found that violating design principles or writing overly complex code is often the stumbling block to achieving good performance." How would you apply this principle today?
Pepperdine: It applies equally today as it did three years ago. Everyone agrees about this now. Brian Goetz and other influential developers have been saying for some time that if you write overly complex code, something's wrong.
I can cite example after example where the act of simplifying code allowed us to get the performance that we needed. Surprisingly enough, this principle actually goes beyond languages, so some of the biggest gains I've seen have been in applications where we simplified Smalltalk code.
In the Java language, it becomes doubly important because now we have runtime optimizations being applied to the code, and complex code confuses the optimizers. When you confuse the optimizers, they do not do a good job of optimizing, so you have a couple of problems with Java when you write complex code.
In fact, if you look at micro-benchmarking, and some of the puzzlers that I do, a lot of the code that I put into the micro-benchmark is there to confuse the JIT (Just-in-Time compiler) so that it doesn't apply the optimizations that it would normally apply, so that the code I've written is what is not actually being measured in the benchmark.
JSC: What are the obstacles to writing more simplified code?
Pepperdine: There are a couple of stressors causing developers to write increasingly complex code.
First, when people don't fully understand a problem, they tend to write code that is overly complex. Second, when they do understand the problem well, they tend to apply a lot of optimizations and simplifications, and when they start out, they apply a lot of extra abstractions to the code and you often end up with messy code.
The problem comes when developers are pressed to go on to the next task, and they never get around to cleaning up the code and end up leaving lots of complex code in their application. The big stressor is not having the time to clean up the code.
When we first started building web applications, we did it the really hard way. Then someone saw an easier way to do it. Finally, we have some really good techniques and languages and tools that help us do it really quickly. It's an evolutionary process.
JSC: As we move toward an era of cloud computing, what should Java developers understand about performance tuning? What scaling problems must developers address these days?
"While there are some problems that are peculiar to cloud computing today, I don't think they will be problems for a long time."
Pepperdine: Cloud computing doesn't look that different from large data centers, which people have been trying to deal with for a long time.
While there are some problems that are peculiar to cloud computing today, I don't think they will be problems for a long time. Everyone knows that virtualization of the network is problematic. People are working hard to find ways to eliminate the problem.
In terms of performance tuning, the tool sets are disappointingly poor for doing diagnostic work in cloud environments or large deployment environments. We need alternative techniques for figuring out what is going on in the system and how to make things more scalable. I'm confident that we will develop those techniques.
All the variability in cloud computing makes it more difficult to define performance problems. You cannot go to your manager and say that you will find the problem by Tuesday and fix it by Thursday — which you can do with a number of problems. Memory leaks are easy to find outside of the cloud, so we can be confident about finding them relatively quickly.
JSC: What are you looking forward to in JDK 7?
Pepperdine: I'm looking forward to more concurrency in JDK 7 and better ways of managing concurrency in the multicore world. The multicore world is changing the landscape and putting a lot of pressures on applications and development. We need more concurrency, especially if Java is going to continue to be the platform of choice.
Java Performance Tuning: A Conversation With Java Champion Kirk Pepperdine
Kirk Pepperdine: Kodewerk.com
Java Performance Tuning
Forum on Java Performance
Java Performance Resources for Java SE 6 and J2SE 5.0 and Earlier
Concurrency Utilities in Java SE 6 and J2SE 5.0
JDK 7 Project
Java Performance Tuning Workshop With Java Champion Kirk Pepperdine