I spent a lot of time towards the end of the past year and the beginning of this year on performance optimization of some Java based server code. I took some notes along the way and wanted to share the lessons learned in no particular order.
Lesson: Know your scenario and focus. There are many uses cases every application needs to support and ideally they all perform as well as possible but realistically, this does not happen, so you need to find the scenario that needs to be optimized, implement a test case that supports that scenario, and focus optimizing that use case. Otherwise, it’s easy to get lost and go in circles while trying to optimize bunch of different use cases. You also need to make sure your scenario is real though. For example, if you need to support a 3 day long scenario, your test should run perform well for 3 days, don’t get too excited that your test ran well for a few hours.
Lesson: Know what needs to be optimized. A lot of metrics come under the umbrella of performance and you need to find out which ones you care about. Is it throughput, latency, number of concurrent users, CPU, memory, or some crazy combination? Once you identify what is important, you need to keep track of all of them in your tests. One tweak might help for latency but it might hurt throughput, or it might lead to high memory usage. Without keeping track of these metrics all at once, you never know for sure whether you met the performance requirements or not.
Lesson: Profiling tools are your friends. There’s no way you can detect thread blocks or CPU spikes by just looking at the code or running tests blindly. You need tools to profile your app, keep stats and report. YourKit and VisualVM are the two I used but there are many out there. Make sure you invest some time in learning them because they really help a lot when you actually need them. I think every developer should be required to know how to use something like YourKit and use it as part of their development to make sure the feature they signed off not only works well but performs well as well.
Lesson: Know your JVM. Most developers take JVM granted and pay little attention to JVM internals such as JVM heap details, garbage collection details, etc. It’s good that we don’t have to worry about JVM internals every day but when your application stops and latency goes up 10 times due to JVM garbage collection kicking in, you gotta start paying attention to JVM details. Changing a few JVM options, such as heap size, or the garbage collection algorithms, can make a huge difference. It’s wise to invest some time in learning JVM internals, so when the time comes for optimizations, you already know what you need to know to pick the right JVM options for your scenario.
Lesson: Test on the right environment. Make sure you’re testing on the right hardware, JVM, app server etc. Performance can vary a lot between different hardware, JVM, app servers. One test case might produce horrible results on your laptop but it might run just fine on a high-end machine, so don’t waste your time optimizing on your laptop when it won’t matter on a high-end machine.
Lesson: Be patient, systematic, and positive. Performance optimization needs patience and absolutely requires a systematic approach. You need to have a good test case, run it through profiler, get the results, identify the bottleneck whatever it is, make changes to get rid of the bottleneck, run the profiler again to see that it helped, and repeat the whole process for the next bottleneck. Same goes for JVM options or any other options you have in your test case. You need to change one thing at a time, let the results guide you in the right direction, and make sure you document your results along the way to avoid going in circles. Most importantly, you need to have a positive attitude because if you’re doing performance optimizations, something must be wrong, and without a positive attitude and patience, it’s very easy to get distracted with the negative results.