How is your Agile or Scrum project going? Is it truly delivering agility to your organisation in the way that the Agile Manifesto or Scrum Guide intends?
In my experience, most Agile and Scrum projects in New Zealand are often not achieving the expected results and in this article I explain the biggest reason why and how it can be fixed.
Agile in a Waterfall sandwich
Walking past my desk today I glanced down and my eyes fixed on the top of my current stack of books. It made me smile, and then I realised what I was seeing and how, purely by chance, the arrangement of books was a metaphor for my working life.
You’ll see the stack in the picture above. There are five of what I consider some of the most important agile books out there, sandwiched between PMBOK and PRINCE2 (the two most dominant flavours of conventional project management). This is pretty much my daily existence, trying to manage the interface between agile product delivery and expectations of predictive, planful governance, and I think I’m not alone in that regard!
This metaphor summarises the challenge that may be affecting your Agile or Scrum project. Let’s unpack what I mean…
The critical role of the Scrum Product Owner
When it’s working well Scrum does an amazing job of removing complexity from the development effort, but it does not make it vanish. It places an enormous amount of responsibility on the Scrum Product Owner.
The Product Owner is aware of the difficulties the team faces because they are there talking with the team every sprint and they help the team by making quick return on investment decisions as required.
The Product Owner must gather up the “rays” of incoming and often conflicting stakeholder expectations, issues around organisational prioritisation ,budgeting, resourcing, etc. and focus them in, once per sprint, to give the development team a clear “beam of light” to follow. The Product Owner role is a little bit like a magnifying glass, or a laser.
Most Scrum Product Owners are unicorns
By unicorn I mean a mythical animal that doesn’t really exist.
Most organisations in New Zealand are too small to have long-term product teams under a Product Owner. More often Scrum is used in a project context and the Product Owner will be the business owner of the capability enabled by the IT change being made. They will also have business as usual responsibilities and, typically, wider responsibilities as a senior manager in terms of supporting other initiatives outside their business unit. Often they simply do not have the time to be the Product Owner as described in the Scrum Guide and this is where the house of cards can begin topple.
Without a fully engaged investor in the form of a Product Owner, you end up with delegates (trusted SMEs or even a Project Manager) and then some form of reporting and change control process where the investor gets information and makes decisions. Enter stage right “Agile in Waterfall sandwich”.
Inadequate Product Ownership leads to the Agile gap
In my last blog post Take a learning journey to agile project management I referred to the ‘Agile gap’, which was my way of describing what I saw as the space between agile as defined by the Scrum Guide and the expectations I am faced with very often working in organisations that, because they have no representative on the ground on the project, revert to wanting formality in terms of certainty around what they will get, when they will get it, and how much it will cost.
As soon as you move out of the space where the investor is making the decisions based on their “on the ground” understanding you move into a space of more detailed plans, status reporting, and steering committees. In essence you have switched control frameworks, from Agile to conventional project management. You are replacing directly experienced understanding and decision making with remote understanding conferred by reports or second-hand verbal information. At the same time you do not have the accompanying project control mechanisms of an approved set of requirements and a detailed project plan based upon them.
As a sidenote, I’m also faced with development teams that often don’t want to acknowledge or understand the realities of a wider organisation outside of the “Scrum bubble”. I’m using this term deliberately and perhaps provocatively, and this is probably an area worth elaborating on another day.
Reality is that today our Scrum initiatives are being torn between agility at the project level and the expectation of certainty at the organisation level. This is akin to attempting surfing on an ocean made of concrete.
Scrum works because it is not expecting 100% certainty
Borrowing terms from industrial process control (which influenced early agile thinking), Scrum is a form of empirical process control, relying on transparency, frequent inspection, and adaptation. This is the preferred approach for when you cannot set up a defined process control model of tightly controlled inputs and processes that produce outputs within tight tolerances (i.e. industrial assembly lines).
Most knowledge work is better managed via an empirical model, simply because it relies on analysis and judgement rather than following pre-defined processes. It simply isn’t realistic to expect assembly line defect rates from a bespoke project that in many cases is breaking new ground. Yet this certainty is what our organisations try to enforce.
Scrum works because it is not expecting 100% certainty. It is based on doing a small batch of work, inspecting, and then correcting. This is what Scrum is. Each sprint, and in fact each daily stand up, is an iteration of inspect and adapt.
The unfortunate result
The result of not having sufficient involvement of a Product Owner, and the following Agile in a Waterfall sandwich, is a project that tries to operate in an agile way but ultimately cannot deliver the desired agility to the organisation. The pressures to deliver certainty to the organisation undermine the intent of the agile project practices and in time the project still refers to itself as agile but in reality is delivering within the undesirable constraints of a waterfall project. The project may still succeed or fail but will never deliver the full potential of value and agility to the business that a pure Scrum or Agile project could have.
Overcoming the challenge
Organisations who genuinely want to follow a Scrum or Agile approach must commit to ensuring that the Product Owner job is fulfilled in its entirety. The Scrum Product Owner must have the trust of the business so that there is no need for formal and certainty-seeking PMI or PRINCE2 governance processes that destroy the potential of the Agile or Scrum project.
It is really important to understand that “being” or “doing” agile is not something that impacts technical staff alone. Adopting an agile delivery method has across the board impacts, most notably a sustained and quite heavy demand for business input all the way through the project. This will in turn impact how the organisation does budgeting and resource planning, and may limit the number of concurrent projects an organisation is able to support.
The rewards of agile are higher transparency, the ability to make your own calls about what is in and out of scope, and being able to determine what a quality product is (i.e. not having the team reduce quality to meet an arbitrarily fixed deadline, which often happens on Waterfall projects when time is tight).
Ultimately agile puts the business in the driver’s seat in a much more direct way than ever before, but it does require a different way of working and a commitment from the business to invest the time and own the decisions.
You can find out more about the Product Owner role by attending our Certified Scrum Product Owner training course.
Carl Weller is a Principal Consultant specialising in Project Management and Agile Leadership, based in our Wellington office.