Is agile dead?

[In this post, I’ll refer to formal processes and frameworks as “Agile”, while using the lowercase “agile” to refer to practices that in my opinion really emphasize agility.]

I’ve been moving the past couple of weeks, so missed the fact that the Agile Alliance has now formally merged with the Project Management Institute. It’s not unexpected, but it does lead me to wonder what has happened to agile? Some of the commentary I’ve seen suggests that this is another indication that “agile is dead” and others have been bemoaning that agile has failed.

While I see the point, I’ll offer a slightly different opinion: agile has proved it works when the conditions for it are ripe, but isn’t ideal for all circumstances. Circumstances have changed in the past 2-3 years, and the place of agile in our portfolio of tools is adjusting, as it will surely adjust again in the future. The current alignment of Agile-promoting organizations merely reflects that change.

I attended my first official PMI event in many years in November before the announcement was made. But even then I was amused at how much the organization that was formerly dismissive of anything that didn’t follow their traditional project management approach, has become so interested in promoting Agile, to the point of coming up with their own Agile approaches and training. I didn’t spend a lot of time on this, because as a formerly certified scrum master (no longer, because I can’t be bothered with the ongoing thousands of dollars to maintain the certification), I didn’t feel it was the best use of my time at that conference.

What has happened to agile methods, and I believe the reason that the Agile Alliance decided to fold itself into PMI is a combination of a few things:

First, the original agile manifesto is a somewhat philosophical document, not any kind of working framework. It allows for a lot of interpretation and flexibility. The people who wrote it came from a variety of backgrounds and favored different approaches. That’s a hard thing for an organization to promote. Or, more accurately, it’s the kind of thing for which a formal organization is somewhat superfluous. For an organization to exist, it has to have something to promote that somebody is willing to pay money for. Once an organization existed, it had to have formal approaches and processes to promote.

Second, the original notion of agility did not appreciate the scale of work that would be taken on by tech organizations over the course of the decades to follow. It was written for developers and system users — not to speak to business management — and it assumed that most of the work could be done in small, independent, co-located teams. It is not a business methodology to coordinate work across dozens or hundreds of teams, spread around the world, all working on related goals that can’t be achieved independently of each other.

Jeff Bezos, arguably, solved this problem more effectively than any person in history, with his now legendary API Mandate, that effectively dictated teams must operate independently. (It concluded, “anybody who does not do this will be fired.”) In a sense, Bezos did an end-run around complex frameworks and the multiple layers of management they expect, by insisting that all work be done in small pieces, by “two pizza teams” that could be agile. Today much of AWS still operates that way at the grassroots level, but even at Amazon, and even in some corners of AWS, tightly-coupled architectures requiring significant coordination still exist, mostly because they sometimes make sense.

Third is that the tech environment has changed. The default in many ways is far more agile than it ever was before. With CI/CD tools and automated testing now the norm, a philosophy of “deliver working software regularly” is about as helpful as a philosophy of regularly eating and breathing. It’s something that everybody just does.

Finally, tech is now a much larger portion of every company’s spend than it was at the time the manifesto was written. This inevitably brings more scrutiny and controls. As I said in my presentation at SCaLE last year, we moved from “move fast and break things” to “year of efficiency” as the corporate norms in the space of 18-24 months. Somewhat anarchic agile ideals no longer fit the times.

The beancounters have — at least for now — won the battle. Agile as we used to know it was best suited to an easy-money, fast-growth culture. In that environment, it’s possible to convince management that quick iterations with no detailed plan is possible, and that in fact it’s the only way for real innovation to happen. This isn’t just me the tech nerd saying so, but even veteran marketer Rory Sutherland is quite adamant that “about 90% of the success in business… is actually highly messy, nonlinear, nondirectional. Most of the effort in business is devoted to pretending that’s not true.“

Rory goes on to bemoan the fact that in the modern world, we are often less interested in solving problems than in winning arguments, often arguments with finance! Agile is terrible at winning those arguments, because “We will increase the likelihood that we come up with a solution you couldn’t imagine, to a problem you didn’t know you had” is not something that can be argued and won “on the numbers.” Agile, as originally conceived, is great at expanding the solution space (something Rory favors), but it’s not as good at predictably delivering on a pre-ordained solution.

As project and program managers, we often find ourselves caught in between, knowing that what needs to happen beneath the surface is iterative and experimental, while needing to explain it in a business framework that can be reported upwards as linear progress towards a stated goal. When you look at Scaled Agile, or any of the other Agile frameworks that are available for sale at great cost from a variety of consultants, they generally exist to allow those two spaces to coexist.

Frameworks and “knowledge bases” that help draw a structure around that messiness are helpful to us in presenting the data upwards. Much of project management has always been about doing this. It’s why ultimately Agile and Project Management end up lumped together even as they emerge from two very different approaches to addressing problems, not unlike the ones Adam Savage describes:

“Given the same amount of time, Jamie will spend 70% of his time on drawings, and end up with something that works; and I’ll use that time to build the thing four times, but I’ll end up with one that works.”

Of the two, Jamie’s “plan, then execute” is far more palatable to most management, but it is only applicable in situations where the desired outcome is precisely knowable before you try. The far messier “try it four times, learn along the way, and end up with something that is suitable for purpose but possibly not what you would have imagined up-front” is far more likely to innovate and push the business forward. But it’s a lot harder to convince management to allow that kind of free thinking, especially when money is tight.

I’ve seen this before. We didn’t call it Agile then, but my early years were very much aligned with the agile principles. Then things matured, controls were imposed, and those quaint ways of doing things were revisited. They would be revisited again, in the opposite direction, eventually resulting in what we now know as agile. I’ve been through this cycle three times in my career.

Agile is in favor, and the organizations that promote it are appropriately consolidating. But it’s unlikely to be the end. Something else will come along, we’ll call it something else, and move forward.