The problem with optimising the software development process

Once upon a time, before I became a Development Manager, I was a developer and, frankly, a technically pretty capable one. I was one of those guys that the other students on my university course relied on to do the team coursework.

I learnt then and reinforced later in my development career the habit of ensuring I did the work required of me as efficiently as​​​ possible. Basically, it meant that I got to go to the Student Union bar earlier. Then later in my career, slightly less selfishly, it meant that I got more work done, the customer got better value for money, and the customer was therefore happier. This makes perfect sense and it's the argument I've had played back to me again and again ever since by other developers, testers, business analysts, project managers and pretty much any other role in the development industry. The problem is that I was wrong.

I was wrong on pretty much every level and I'm going to split up the above statements to point them all out.

"I did the work required of me as efficiently as possible"

Once I'd sorted out the obvious efficiencies in my tools and working environment, like IDE shortcuts, templates and code generation, the major efficiency I could gain was by grouping related work. For example, work that was in the same area of the code base, or needed the same tools, or other sensible reasons like that. Looking back I can see that I recognised, correctly, that context switching was expensive and that I should avoid it. I know this served me well because I got to the student bar most nights. This is a lesson that most developers seem to understand intuitively.

So why is this part of the statement wrong? It's wrong because while I was making myself more efficient at producing code, that wasn't actually the work required of me. What was required of me was producing something of value for a customer. At university I may have been right because it really was a lot more about producing code than producing value so perhaps I can blame the education system for this! However, the process of producing something valuable for a customer involves an awful lot more than just writing code. Essentially I got fantastic at optimising locally at the expense of optimising the process as a whole. Did I do my work efficiently? Absolutely. Unfortunately in doing so, I was making the rest of the process less efficient. Read on to find out why!

"... it meant that I got more work done"

Don't get me wrong, I was definitely producing more code, but the meaning of "more work" in this context is so much wider than that.

Now this is moving more firmly to where my interests lie these days in the overall development management space. The process of producing value overall varies from company to company based on what the responsibilities of the company are to the customer. Let's look at two common scenarios.

Scenario 1 is a development shop which is contracted by a customer to run a project to produce a product. There are obviously a number of options and variations here but fundamentally the process as a whole we want to optimise includes not only the development process (development, testing, deployment, documenting etc.) but also the process of ensuring that we are producing the right product for the customer. This means that the requirements gathering, user acceptance and feedback processes are also all part of the whole.

The problem with my grouping or batching up the work needed of me as a developer is that although I had reduced the overall time it would take for me to do the software development, I was actually taking a lot longer to produce any single feature. This meant that the testers were waiting longer to get my code and ultimately the customer was waiting longer to see the next feature. As a result the customer didn't get to provide feedback as often, and when they did, it was more extreme in both volume of change and reaction. A lot would have changed in the extended timeframe and the functionality delivered could be a long way from their vision, irrespective of the requirements they'd provided me. Another side effect of the additional delay in feedback was that there was significantly less flexibility to make changes because there wasn't the time in the schedule overall to do it.

Scenario 2 is an IT department which is responsible for software development and support of the products they develop. On face value this is very similar to Scenario 1 when considering the process. The significant difference is the need to support the product. Even when only level 3 support (i.e. only defect investigation and fixing) is provided, if the quality of the product produced by the software development team isn't incredibly high, then failure demand will start to impact team performance. This may impact the rate at which the department can produce value in the form of new software development, or worse it will impact the quality of future products because corners are cut to keep the value production rate up. The reason the second option is worse is that it will snowball very quickly to the point where the department is forever investigating and fixing defects and never producing anything of value anymore.

The point of covering the two scenarios above is to demonstrate how far-reaching the overall process can be and the huge number of variables that form part of the problem. The process we need to optimise overall can be huge and not considering a factor from the overall process when locally optimising can have a huge impact on the whole.

I now understand why optimising my efficiency as a developer in the form of batching work was negatively impacting the overall process of producing value for the customer. This is essentially one of the core arguments for optimising on cycle-time or throughput, for the whole process, ahead of optimising for utilisation of individuals.

"…. the customer got better value for money"

Looking at everything we've covered so far, I'm sure you can see that while I was definitely the most efficient cogwheel in the engine, I wasn't actually personally generating better value for the customer and in fact, because I was so personally efficient, the engine itself was actually producing worse value for money for the customer.

"… the customer was therefore happier"

And finally, over many customer engagements whether they've been internal to my organisation at the time or not, I finally realised that in the end the customer doesn't really care about value for money, at least not once they've chosen me as the provider and concluded the commercial negotiations. They've already decided how much they're willing to pay for the service of producing what they've asked for. Once the process has started, what they really care about in my experience is seeing their vision realised and being involved in the realisation of it. So that's what I now optimise for - producing happy customers.​​