Unnecessary complexity, why does it happen?

In software industry, there’s a strange disease called complexity. Especially complexity that exists for no good reason. I don’t know why it happens exactly but I know it’s widespread from experience. Software development is a complex process and the problems we deal with are somewhat complex as well. But these problems are hardly rocket science. They are nothing compared to space exploration or cancer research, for example.

Yet, every day, I see unnecessary complexity scattered around software projects. It’s usually a hassle to read someone else’s code. Trying to use a new framework is a steep learning curve. A new programming language? That’s even worse. Something that intuitively feels like it should be simple, usually ends up being quite complex to implement. Why is that? I don’t know the exact answer but I have some ideas.

First has to do with the people involved in software development. These are usually smart, motivated people who like challenges and puzzles. Otherwise, they wouldn’t be in the business of writing thousands and thousands of lines of code in-front of a screen, day in and day out. Smart people have the tendency to over think, over engineer, over analyze. While regular people have difficulty in grasping complexity, these people have difficulty in appreciating simplicity. When you have so many smart people around working on the same problem, complexity arises naturally and simple solution are often forgotten or ignored.

Second has to do with human nature. People are social animals with egos and they like to assert their dominance. In animals, dominance is often asserted with raw physical power whereas in humans, it’s more about intellectual power, at least in software development. In a software project, whoever shows the mastery of complex problems and solutions is often assumed to be a competent person. In design meetings, everyone loves to yell out complex ideas and buzz words to assert their competence within the group. While mastery of complex concepts is definetely a good attribute, a better attribute is to have the ability to find the simplest solution to a given problem but the latter is often ignored because it doesn’t help in asserting dominance.

Third has to do with the overall lack of discipline in software design process. While designing software systems, there are a lot of moving parts, a lot of people involved and there’s limited time and patience. The whole process is often rushed due to time and patience constraints. In that environment, it’s easier to find a “good-enough” solution and move on. That works for part of the problem but when you add many “good-enough” solutions, you usually end up with an over-engineered, over-complex system overall. There’s most definitely a simpler and more elegant solution but it requires a more disciplined and more rigorous design process that hardly exists in a typical software shop.

Given all this, simple problems that should have simple solutions end up with unnecessarily complex solutions. Complexity is like a virus. Once you have it in an inner layer of the system, all the outer layers eventually suffer from the same problem. Pretty soon, you have a overly complex system that nobody knows how to maintain, every change causes a ton of other changes. Unnecessary complexity turns the optimistic, positive, creative process of software development into a fearful, dull and frustrating process both for creators, testers and users.

All of this happens because we didn’t take the time to find a simple solution and we didn’t resist the waves of complexity during software design. Next time you design a piece of software, make sure you spend some time to find a simple solution. A solution that is easy to grasb, easy to explain, easy to implement and easy to test. Such a solution exists, you just need to spend the time to look for it and you need to have to courage and will to resist the evil of unnecessary complexity. The success of your projects depends on this.

Accountability and Sense of Ownership in Software Development

One of the most overlooked concepts in software development is accountability. Accountability means a team is totally responsible for successful implementation and execution of a piece of software or a service from the beginning to the end. From the time the software/service is envisioned and designed, to the time the software/service implemented and gets used by the end users, a single group of people is involved in the whole process and once it goes out of the door, that team is held accountable for the successes and failures for the end result.

This is the only way to make sure there’s a sense of ownership in a software project. Sense of ownership ensures that people in a project feel some kind of emotional attachment and responsibility towards the end result. Without a healthy dose of sense of ownership, it’s very easy for teams to develop an “ain’t my problem” kind of attitude and that’s not healthy to have in any project.

In big software companies, priorities often change and as a result, some teams are asked to stop working on a certain project, transition the project to some other team and start another new project. To upper level management, this is not a big deal, they’re just moving “resources” around from one project to another. But anyone who has done some kind of software development knows that this is much more than simply moving resources around.

By switching teams and projects around, accountability and sense of ownership in a software project are ruined. The team leaving the project will not care about the project or held accountable for it anymore because they will have a new project to work on. The team inheriting the project cannot be held accountable or feel ownership for the project either because they were not involved from the initial design. In the end, customers suffer due to poor quality. After a few project shuffles, people in the company gradually stops caring about accountability or sense of ownership because they expect that sooner or later their current project will be reshuffled and they are not going to be accountable for that nasty production performance bug.

To sum up, if someone asks you how to make sure a software project fails? Quite easy. Assign it to one team, then reassign it to another team in the middle. In the end, you’ll have an orphan piece of software that nobody feels responsible or accountable for and you planted the seeds of a non-caring unaccountable culture. Sure way of killing current and future software projects.

Team formation: another broken process in software development

In my previous post, I talked about why I think the technical interview process is totally broken. In this post, I want to talk about yet another broken process in software industry: how teams are assembled (or more like misassembled!) for a software project.

In a typical software company, a person is hired by a team to work on a particular project within that team. This makes total sense. However, after some time, things change. The actual time frame varies from place to place, it could be as soon as 1 month or something longer like 1 year or more. Either way, change is inevitable, the project either ends or completely changes or the team gets a completely new assignment. This is the point where it gets really weird. Now, the person who was hired to work on a particular project is asked to work on a completely new project. The hired person is basically stuck with the team he/she was hired into to work on a completely different project that he/she did not sign up for.

If the project and the team sound interesting, he/she can carry on and all is good. However, the problem arises when either the project is not interesting or does not align with the person’s goals or the team is not what the person hoped to be. At this point, the hired person has a few choices but none of them are good. He/she can try to get out of the project/team but that means he/she needs to find something else to do within the company. In big companies, there’s a lot bureaucracy around switching to a new group. You usually need to get your manager’s approval (which is quite awkward as it is) and then go through the process of making yourself accepted in the new group. It’s just not straightforward as it should be.

The other choice is to quit the company and join to some other company which is even a bigger deal with more interviews and process. Either way, you end up spending a lot of time not doing any real work. So, what usually happens is that, many people go on, accept their faith and continue working on uninteresting projects with uninspiring people because it’s easier for the time being. This naturally results in uninspired mediocre work and it goes on until either the company or the person realizes how ridiculous the whole situation is. The person either quits or gets fired.

I’ve seen this happen a few times to solid people and it is quite sad. It does not happen because person is incompetent but it rather happens because there is a misalignment between the person’s abilities and his/her place in the company. And there isn’t a good and open process to fix that alignment within the company. This is especially true in big companies.

I think the solution is not that difficult though. In big companies, there should not be strict hierarchies with solid lines between managers and developers. The weird notion that a single manager effectively owning a software engineer within a particular group should change. Instead, software engineers should be treated more like free agents. They should be assigned to projects for a period of time and when that time elapses, there should be an opportunity for an open discussion among the employee, manager and possibly someone from HR to figure out what makes sense as the next step. Software engineers should be presented with project choices within the company at that point in time and they should have the real option of staying in the current group with the current project or move to another project with little or no process. The company hierarchy should not dictate what people do and where. Rather, the business needs should dictate what the projects should be and then people should be able to gather around those projects freely, kind of like open source project model but within an organization.

Big companies will tell you that they already encourage people to move around within the company but in reality, this is far from true. First, the amount of process and bureaucracy involved is so high that many people do not even bother. Second, the culture at these companies does not encourage moving around. The very existence of a strict hierarchy dictates the boundaries that an engineer needs to adhere to and it is quite difficult to get out of those boundaries. Third, a software engineer is not presented with other choices once he/she is hired into a role and there is hardly ever an open communication about the employee’s next step. Employees end up sneaking around the company hierarchy to find the right alignment and it just should not be like that.

Open source software is so successful mainly due to people involved in those projects. Most of open source people put time and effort into those projects because they really care about them, not because of a random hierarchy. The same should happen in software companies as well. The company that figures out a way to let people gather around projects they care about voluntarily will create the winning innovative culture that is so lacking in big companies nowadays.