Performance can make or break a piece of software, this is clear; nobody puts up with an unresponsive client UI or a slow back-end server in today’s age of software abundance and choice. Despite this, performance is often overlooked until late in the release cycle and doesn’t get the proper attention it deserves. This might not be a big deal in one release cycle, but after a few release cycles, you can end up with a slow-moving giant that nobody knows how to fix instead of a lean fast machine that you used to have. At this point, you either accept what you have or you take the hit and go through the painful process of profiling, analyzing, fixing, and in some cases redesigning. How does software end up like this in the first place? I can think of four reasons.
First, it’s not easy to sell performance. New features, especially visual ones where people can see and play, often are much easier to market and sell then subtle yet more important features like performance. Two more features on the release cycle looks better than 20% increase in throughput for example, so performance is not treated as a proper feature but rather seen as a thing to check at the end of the release cycle. As a result, performance does not get the time and resources it needs.
Second, it’s not easy to reason about performance. You need to define what metrics are being measured in the name of performance, define what qualifies as acceptable performance, define use cases where performance is important. This requires through understanding of the software and the use cases around it. It’s hard to get the scope of performance work right, it’s either too broad to implement or too narrow to produce anything useful.
Third, performance work is hard, sometimes harder than implementing the software. It usually needs additional tools/software outside of the software itself in order to write tests to simulate the agreed upon use cases and track some numbers around those cases. In most places, there is simply not enough time left over outside the feature development to build those tools. Even if you have all these tools, you need time to run the complicated performance scenarios and if numbers don’t look right, you need time to find out why; it can be anywhere in the code. You also need to do this all over again in every release cycle or get some time to implement automated performance tests that can track performance for you. This is a lot of work.
Fourth, performance is usually not tightly integrated with the overall feature development. When a new feature is being developed, there is a lot of focus on the new capabilities that the new feature brings from Engineering, QA, Product Management but not as much focus on two things: 1. How does this feature perform by itself? 2. How does this feature affect the overall performance? The result of ignoring #1 is that a new feature gets designed and developed without performance in mind, and by ignoring #2 the overall performance of existing system gets worse which is even worse.
Despite all this, professional software developers have the obligation to design and implement performant software, no matter what the realities of the workplace is and I think with some effort, performance can be saved and maintained over the release cycles with a few guidelines that I hope to share in a future post.