A very large, very competent financial services company asked me to help them understand why their software delivery dates were constantly slipping, and what to do about it. They used agile methods to run their projects. I expected to spend months looking through process documents, consulting with business partners, attending project meetings. I hardly did any of that.
It didn’t take me long to figure out the problem: they had completely misunderstood the purpose of agile methods. This caused the company to incentivize the very problems agile was supposed to cure.
You might think that was a good thing for me. But the unwritten and informal incentives had career-making and career-ending implications for many powerful interests in the company. As a consultant, even approaching such a change is scary and risky.
I did it anyway. It was messy and excessively contentious at times, but I did change the incentives and return their Program Management Office back to the black. To accomplish this, I had to overcome some mythology about agile methods.
Before you read the rest of this article, it is probably best to consult the original ideas leading to agile methods. The Agile Manifesto, which lays out purposes, but what interests here are the values they sought to establish.
- Individuals and interactions over processes and tools.
- Working software over comprehensive documentation.
- Customer collaboration over contract negotiation.
- Responding to change over following a plan.
It’s that last one that seems to be grossly misunderstood, while the first three are often undermined by informal incentives. I like to approach most problems genealogically. As G.K. Chesterton observed:
In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”
Chesterton wants us first to observe a difference between reforming and deforming. We reform when we alter the purposes of a thing to better ends. We deform a thing when we merely change its form, methods, or practices without regard to the ends it serves.
Once we know the purpose of a thing, and only when we know the purpose, we can then evaluate whether those purposes serve us adequately. I find the best way to understand the real purpose of a thing is to understand how it got there in the first place.
And here we must distinguish between real purposes and nominal purposes. It is very common in large organizations to have two worlds functioning in the same space. There is the world designed on paper, the nominal organization, and the world as it really is, the real organization.
Agile is prone to misuse by conforming to the nominal organization, while undermining the real one. To save my client’s commitments to investors, I had to clear away mythological beliefs about agile and steer the real organization towards real agile. Here are some of the myths I presented to my client.
Myth 1: User Stories Generate Good Designs
Code does not have a tendency to self organize. It’s quite the opposite. This fact is contrary to the assumptions of most agile certification programs.
Code suffers from increasing entropy as as the codebase evolves and is adapted to customer needs. When we add new functionality, the system becomes more complex. Often it also becomes more complicated.
An adaptable, coherent system design is very unlikely to materialize from independent user stories, even if we refactor out the commonalities.
Reality: agile is not an excuse to avoid up-front architectural design. Agile architectures are created up-front with extension mechanisms to adapt easily and rapidly. Agile is good for coding but not for design.
Imperative: design good, extendable architectures first. Design requires long-term thinking about how to use short-term development schedules.
Myth 2: Agile Processes Assure Agility
Agility is the ability to understand and change quickly. Agility is the result of good designs, especially architecture. Agility is not the result of organizational processes.
I walked my client through a painful reevaluation of their design processes. Some of the things we discovered together:
- Most of the design time was spent drawing burn-down charts, instead of design diagrams.
- Project managers spent much time on estimating user stories, instead of letting the developers work through the algorithm complexity.
- Agile team members spent lots of time tracking their velocity, instead of following the latency and throughput of their software.
- Teams spent lots of time in meetings and retrospectives, but little time in design and code reviews.
The pattern is clear. The process was prioritized over the product – in complete contravention of agile purposes and values.
Reality: agile processes can help or hinder real agility. It depends on how they are purposed and incentivized.
Imperative: focus on the system not the process. Agile processes must serve the purpose of building flexible and adaptable software, not the other way around.
Myth 3: Constant Refactoring is the Best Way to Build Software
Constant refactoring incentivizes poor design up-front. Developers focused on their velocity measure will not plan for change, because they believe it’s always easy to refactor the system.
Ironically, the classic symptom of this problem is the velocity measure itself. Teams will initially implement features quickly with little up-front design. As they adapt the poorly-designed system, their velocity slows in proportion to code complexity.
Reality: refactoring incentivizes increased entropy in the codebase, rather than preventing or decreasing it.
Imperative: plan for change up-front in the architecture, and in each testable unit of software delivery too.
Myth 4: You Can Pay Technical Debt in the Future
Yeah, people say that about credit card debt too. You can pay the principle, but you can’t pay the interest. In a complex software system, the debt is the change you need to make. The interest is all the other changes to other people’s code that makes it all work.
You’re going to face the N2 problem.
Reality: you cannot refactor your way out of technical debt. The interest is too high.
Imperative: follow your mother’s advice, don’t accumulate debt in the first place. Pay your way, as you go, with up-front extensible architectures and adaptable designs.