Latest LCDS Performance Brief

Latest LCDS Performance Brief has been published here (6.8MB). One notable addition is the new information on remoting performance in LCDS.

 

Performance and heap size

You have a machine with lots of memory, your Java/J2EE application is the only one on that machine, and you want high throughput and low latency from your application. I can guarantee that you’ll need to make some decisions on the size of the heap to allocate for your application at some point (using -Xms and -Xmx flags).

You can start with no min/max heap size, let JVM start with a default min heap for your platform and adjust heap size as needed. My observation is that adjusting heap size is expensive and affects performance, especially if your application is running under load. You can choose a small min heap, so at least the initial heap is chosen by you rather than JVM, and a larger max heap but this doesn’t buy you much because if your application needs more memory than min heap, JVM needs to adjust the heap size again and it’s the same performance problem as before. You can choose a largish min heap (something more than your application will need) and an equal or larger max heap. This sounds like a good idea at first, JVM with plenty of memory to work with should not cause any problems, right? Wrong. When more heap is allocated than what the application needs, garbage collection (GC) seems to get lazy. Of course, GC can be tuned with other JVM options but generally large heap results in lazy GC because JVM can afford to be lazy when it comes to GC in high throughput scenarios. When GC is lazy, minor GC collections kick in less frequently and eventually turn into full GC collections. Anyone who briefly looked into GC details (using -verbose:gc and -XX:+PrintGCDetails options) would know that full GC takes a while and it definitely affects the latency of your application as it’s running.

So, what can you do? Setting the min/max heap to what your application actually needs is an obvious first step. You need to find out how much memory your application really needs, you can do that but running your application through profiler (eg. YourKit, VisualVM). Once you have that value, set your min and max heap slightly higher than that value. By doing that, you remove the JVM heap sizing out of the equation and since the max heap is a reasonable value (around what the application needs), GC won’t get lazy and full GCs will be less frequent.

Of course, there are many other JVM options that affect performance. Options on which GC algorithm to use such as -XX:+UseSerialGC, -XX:+UseParallelGC, -XX:+UseConcMarkSweepGC, -Xincgc, or options on how many threads GC should use such as -XX:ParallelGCThreads, or how much GC should run such as -XX:GCTimeRatio, -XX:MaxGCPauseMillis all come into play and maybe we’ll cover these in some other post.

 

LiveCycle Data Services Performance Brief

Are you interested in learning more about the performance characteristics of the messaging infrastructure provided by LCDS? If you have performance related questions such as “How does the number of concurrent clients affect latency?” or “How does message size affect latency?” or maybe more fundamental questions like “Why should I care NIO-based endpoints in LCDS?” or “What’s the latency overhead of using LCDS Edge Server?”, then you need to check out the Adobe LiveCycle Data Services ES2 Performance Brief (pdf) on Livecycle Developer Center. Not only it provides answers to a number of questions like these, but it comes with tools and instructions on how to reproduce those results yourself if you don’t believe us

 

Real-time systems with Java

I came across a six-part series of articles from IBM on challenges to using Java to build systems that meet real-time performance requirements. The articles do a great job of outlining the challenges and also introduce IBM’s real-time VM that try to address these problems. Great read for anyone remotely interested in real-time applications.

Real-time Java, Part 1: Using Java code to program real-time systems
Real-time Java, Part 2: Comparing compilation techniques
Real-time Java, Part 3: Threading and synchronization
Real-time Java, Part 4: Real-time garbage collection
Real-time Java, Part 5: Writing and deploying real-time Java applications
Real-time Java, Part 6: Simplifying real-time Java development

 

Performance optimization lessons

I spent a lot of time towards the end of the past year and the beginning of this year on performance optimization of some Java based server code. I took some notes along the way and wanted to share the lessons learned in no particular order.

Lesson: Know your scenario and focus. There are many uses cases every application needs to support and ideally they all perform as well as possible but realistically, this does not happen, so you need to find the scenario that needs to be optimized, implement a test case that supports that scenario, and focus optimizing that use case. Otherwise, it’s easy to get lost and go in circles while trying to optimize bunch of different use cases. You also need to make sure your scenario is real though. For example, if you need to support a 3 day long scenario, your test should run perform well for 3 days, don’t get too excited that your test ran well for a few hours.

Lesson: Know what needs to be optimized. A lot of metrics come under the umbrella of performance and you need to find out which ones you care about. Is it throughput, latency, number of concurrent users, CPU, memory, or some crazy combination? Once you identify what is important, you need to keep track of all of them in your tests. One tweak might help for latency but it might hurt throughput, or it might lead to high memory usage. Without keeping track of these metrics all at once, you never know for sure whether you met the performance requirements or not.

Lesson: Profiling tools are your friends. There’s no way you can detect thread blocks or CPU spikes by just looking at the code or running tests blindly. You need tools to profile your app, keep stats and report. YourKit and VisualVM are the two I used but there are many out there. Make sure you invest some time in learning them because they really help a lot when you actually need them. I think every developer should be required to know how to use something like YourKit and use it as part of their development to make sure the feature they signed off not only works well but performs well as well.

Lesson: Know your JVM. Most developers take JVM granted and pay little attention to JVM internals such as JVM heap details, garbage collection details, etc. It’s good that we don’t have to worry about JVM internals every day but when your application stops and latency goes up 10 times due to JVM garbage collection kicking in, you gotta start paying attention to JVM details. Changing a few JVM options, such as heap size, or the garbage collection algorithms, can make a huge difference. It’s wise to invest some time in learning JVM internals, so when the time comes for optimizations, you already know what you need to know to pick the right JVM options for your scenario.

Lesson: Test on the right environment. Make sure you’re testing on the right hardware, JVM, app server etc. Performance can vary a lot between different hardware, JVM, app servers. One test case might produce horrible results on your laptop but it might run just fine on a high-end machine, so don’t waste your time optimizing on your laptop when it won’t matter on a high-end machine.

Lesson: Be patient, systematic, and positive. Performance optimization needs patience and absolutely requires a systematic approach. You need to have a good test case, run it through profiler, get the results, identify the bottleneck whatever it is, make changes to get rid of the bottleneck, run the profiler again to see that it helped, and repeat the whole process for the next bottleneck. Same goes for JVM options or any other options you have in your test case. You need to change one thing at a time, let the results guide you in the right direction, and make sure you document your results along the way to avoid going in circles. Most importantly, you need to have a positive attitude because if you’re doing performance optimizations, something must be wrong, and without a positive attitude and patience, it’s very easy to get distracted with the negative results.