Do Waterfall and Agile hybrids make sense?

Image: Grist Mill Water Wheel - Sudbury by Massachusetts Office of Travel & Tourism, used under Creative Commons CC BY-ND 2.0.

There have been various opinions and articles lately on the merits of marrying Agile, Scrum and Waterfall approaches to create hybrid models, for example Planning and Managing Development Projects: The Hybrid Way by Michael Wood.

We tend to be very pragmatic at Equinox IT, focusing on what works in practice. We’re the last place you would find uber-enthusiasts, swearing allegiance to one “right” project management or software development framework.

However, that is not to say that we are completely Laissez faire, and some thought does need to go into whether Waterfall and Agile hybrids actually make sense. One of the things that Michael Wood’s article (as well as others linked within that article) seem to miss is that Agile methods use a completely different control framework from Waterfall.

Waterfall uses a ‘defined process control’ framework

Waterfall is based on industrial age approaches and uses the same ‘defined process control’ framework that we would see in assembly line manufacturing. These practices are based on an assumption that all dependencies inherent in the work can be understood and managed. Often there is a substantial investment required to get processes this tightly controlled (e.g. BMW spend 500 million Euros on the assembly line for the new 3-series). The problem is that software development is not generally set up in a way that generates the same output every time in the way that assembly line manufacturing does.

Waterfall may still work for software development, but would be most useful when the problem being solved is obvious (the relationship between cause and effect is obvious to all) or complicated (the relationship between cause and effect can be identified by an expert). An example would be rolling out an off-the-shelf product with very little change from the base configuration.

The terms obvious (simple) and complicated, as well as complex and chaotic come from Dave Snowden’s Cynefin Framework, which you can find more about in his 2007 HBR article A Leader's Framework for Decision Making.

Agile and Scrum use an ‘empirical process control’ framework

Agile and Scrum use an ‘empirical process control’ framework, that exercises control through frequent inspection and adaptation. The process supports uncertainty, where specifications are imperfectly defined and the outputs are not about conformance to standard, but rather changing the plan frequently as more is understood about both the features required and the challenges inherent in building the system. Often people don’t know exactly what they want early in the project, but usually discover it as they go along. Unfortunately, Waterfall projects lock in those things we thought we wanted at the very time when we knew the least about the project.

Scrum and other Agile approaches rely on transparency (e.g. use of big visible charts, an open backlog and open estimation), and inspection and adaptation. In Scrum there are feedback loops between Sprint Planning and Sprint Review that ensure there is high transparency and the plan is adapted using performance data on a regular cadence.

The reason a Sprint is so powerful is because Scrum creates a "container" around the team for that 2-4 week period (where the requirements are pretty well understood and there are no distractions such as change control). The team can actually just get on with it. For more on this topic see my previous article The important role Scrum bubbles play in Agile project success.

I’d argue that software development approaches that support an ‘empirical process control’ framework are increasingly more suitable to today’s world. Many problems we face are not obvious or complicated, but are complex (the relationship between cause and effect can only be identified in retrospect, not predicted in advance). The rapid changing nature of the world means that increasingly software development will be complex, simply because it’s not just the creation of software that holds unknowns, but also the environment we are delivering it into is changing and evolving. We may see projects needing to “pivot” mid-stride to borrow a term from The Lean Startup.

This article from Ken Schwaber is a few years old now, but does a great job at elaborating on what I am talking about - Waterfall, Lean/Kanban, and Scrum.

Does mixing Agile / Scrum and Waterfall make sense?

For me, not really. Let’s assume that your problem space is most likely complex and not obvious or complicated.

Where this is the case, then using standardised techniques, formal analysis and planning approaches from Waterfall is unlikely to be of value. Mixing these practices in with Agile or Scrum practices doesn’t make sense, because the Waterfall practices are planted firmly in ‘defined process control’ which won’t help you run a successful software development project for a complex and unpredictable problem.

I can see that, where your organisation has stringent QA requirements that must be met before releasing to production, a Scrummer-Fall model may be a useful mechanism (i.e. Agile requirements elaboration, build, and testing up to pre-production, with a transition to a final formal testing period before deployment). Given there should have been a high level of engagement with the business, and you should have been using modern technical practices during development, the final testing phase shouldn’t throw up anything major. There are other hybrids out there that are also worth investigating, so long as you are aware of the nature of your problem space.

But for me it’s not so much about ‘are these two approaches compatible?’. More important is ‘how do I help technologists and business people bridge the gap?’. In other words, ‘how do I help people adopt the right approach needed to solve the nature of problem they are facing?’. Here hybrid approaches like PRINCE2 Agile and more bespoke methods may have a role to play. However, care must be taken.

My main issue with the PRINCE2 Agile approach is that, if you look at it in a simplistic way, it treats a Sprint Plan as a Team Plan to deliver a Work Package (i.e. Agile is just a collection of technical practices and a different work allocation model). That's where you can end up with abominations such as Water-Scrum-Fall (formal requirements phase >>> "agile development" >>> formal testing phase). This completely fractures the fast-cycle feedback loop between the team and the investor, destroys any value from ‘empirical process control’, and lands us back in ‘defined process control’. You need to set up the overall PRINCE2 control framework with sufficient “flex” for agility (something PRINCE2 themselves acknowledge).

The pragmatic way forward

There is no one right way, and, as the Cynefin framework itself would say, you need to experiment, amplifying successes (doing more of what works) and dampening failures (doing less of what doesn’t work).

However, if you do try a hybrid approach to project management and software development, which marries together Agile / Scrum and Waterfall approaches, do so with your eyes wide open. Give consideration to what the different frameworks are trying to achieve, and don’t mix things in such a way that you get the worst of both worlds.

For me the difficulty, as always, is not so much the choice of approaches as it is communicating the differences and agreeing a middle ground. I don't have all the answers and I wouldn't believe anyone who said they did!

You may also be interested in my article The Scrum Product Owner challenge that may be affecting your project.

Carl Weller is a Principal Consultant based in Equinox IT's Wellington, New Zealand office.