Sunday, April 22, 2007

Snake-Oil Salesmen, Process Skepticism, and Why I am Not a "Post-Agilist"

The optimist thinks that this is the best of all possible worlds, and the pessimist knows it.
-- J. Robert Oppenheimer, "Bulletin of Atomic Scientists"

While I am a strong proponent of agile methods, I've witnessed a number of people I respect writing about being skeptical of agile processes and complaining about evangelists who try to sell methodology without really knowing what they're talking about. For example, Jonathan Kohl has written about Post-Agilism where he states :
Personally, I like Agile methods, but I have also seen plenty of failures. I've witnessed that merely following a process, no matter how good it is, does not guarantee success. I've also learned that no matter what process you follow, if you don't have skilled people on the team, you are going to find it hard to be successful.

Ravi Mohan has written of Agile as a religion, "with its holy books, many patriarchs, its churches, and commandments." In his blog, Ravi also frequently decries the lack of actual programming experience demonstrated by many self-professed gurus to back up their claims.

On a certain level I agree. It seems evident that any movement, as it grows beyond a certain point, becomes vulnerable to distortion, both by the well meaning but ignorant, and also by those who cynically aim to make a buck from whatever is trendy today. Agile is no exception. One thing I stress whenever I talk about agile methods is that the agile manifesto is self-referencing. The first statement (Ravi would perhaps say commandment!) of the manifesto is clear:

Individuals and interactions over processes and tools

Those "processes and tools" include the methods of Scrum and XP! If you're working on an agile project and the people on the project are trying to blindly follow some agile practice when it clearly doesn't make sense, they are violating the spirit of agile development: Ipso facto, you must use your knowledge of what's actually happening on your particular project, your team, at a particular point in time to decide what to do next. Everything else is just a guideline.

However, things are unfortunately not so simple because it is often not clear when you should abandon a given practice. Let's say you think that something isn't testable and you give up test-first development; then you decide that a given feature really requires a lot of up-front design and you give up iterations. Before long you've entirely given up all the essential aspects of what agile software development is about! Following an agile practice is therefore always a delicate balancing act. You must respond empirically to reality, but you must learn the practical value of agile practices. I've found that many of these practices are worth defending. For example, at times I have been tempted to abandon test-first development when testing became difficult, but upon deeper consideration I realized that there was something wrong with the way I was approaching my problem, and that was the real culprit that made the test-first approach problematic. By re-thinking my approach, I found a way back to test-first development and its benefits.

I guess I would like to emphasize the following point: If you are interested in agile development, you must take the time to learn the value of the practices you're using. Blindly doing anything will never do any good. However, discomfort I have with notion of "post-agilism" is that the malaise people are experiencing, that's causing them to react, is rooted in fundamental problems: Managers will read articles about agile and impose it on their teams; so-called gurus will go into a company dispensing advice without properly understanding what's going on, perhaps without really knowing anything of value in the first place; teams will get caught up in zealous behaviour and start following practices blindly; people will splinter into factions, arguing over minutiae and jealously guarding their particular version of the Truth; incompetence will have its day. Thus dogmatism will set in. The way I see it, these things are inevitable. They are inevitable in the same way that corruption is inevitable in any social or political order. As with corruption, one can only do one's best to manage such problems, to reduce them to a minimum, but one will never get rid of them entirely.

I fear that creating a new movement/idea and calling it "post-agilism" is just sweeping these basic problems under the rug. If the agile movement is a case of taking some pragmatic ideas and codifying them into a general system, or "going meta" with them, as Bob Martin has written, then "post-agilism" is a case of going "meta-meta." I can't see how this will lead to any good. Before long, there will be "post-agilist" gurus and dogma, and because of its second order nature, the attendant problems will be much harder to deal with and untangle. I hope that the agile community will get itself out of this spiral and instead will establish ways to do research and to educate people so that as much as possible software development will be about common sense and about what works instead of dogma. Things will never be perfect, but hopefully we can work toward a system that represents the best possible balance and thus minimizes the inevitable corruption that accompanies any human endeavour.

An agile development process ought to be self-correcting. If that's not working, I'm not sure how embracing "post-agilism" will help: For better and for worse, I think the agile approach represents the best of all possible worlds.


JonathanKohl said...

Hi Vlad;

I appreciate the sentiment in your post, and I like the fact that you are working to defend something that works for you, and that you like.

"I fear that creating a new movement and calling it "post-agilist" is just sweeping these basic problems under the carpet."

I don't see this as creating a new movement - it's a descriptor of an emerging new era. No one created it, it just happened. I first saw this behavior in 2003, and I've seen it grow as more and more people move beyond Agilism. The exciting part of it is that it demonstrates progress. It demonstrates that the Agile movement made its mark.

"I hope that the agile community will get itself out of this spiral and instead will establish ways to do research and to educate people so that as much as possible software development will be about common sense and about what works instead of dogma."

Challenging the hype and poor behavior is one way to help people move out of a vicious circle. In fact, some people have told me that the term "post-agilism" helped them get out of a spiral, move ahead, and improve their software development work, building on the good they learned from Agile methods.

Eventually, I hope we don't need terms like "Agile", "waterfall" etc., we'll just return to trying to create great software. That will foil the marketers and third rate consultants for a while.

You are absolutely right than any movement will get corrupted at some point, which is why some people get tired of the same-old same-old, and try to create something new. What will a new era create that we can benefit from? That's what interests me. I hope we never stop innovating.


Ravi said...

I second what Jonathan said.

Also you need to know that the "post agilist" folks are *also skilled agilists*. They know from personal experience where the practices of agile work and where they don't. This is not a blind lashing out. No one denies that agile can work (and work well) when applied in the appropriate context.

Think of XP (say) as a hammer. There are situations in which its use is appropriate. The disconnect happens when there is a "cult of the hammer" with "Certified Hammer Masters" and arguments like "If you can't use a hammer to unscrew screws it is because you don't know the One True Way of hammer usage".

So, in a nutshell, "post agilists" (note the quotes) are people who "get" agile but aren't blinded by it or depend on other people being blinded by it so they can sell consulting services.

Agile is one possible method of building software. Nothing more and nothing less.

Ravi said...

ON second reading, think there is an even more fundamental problem. You assume that "post agile" is a reaction against the *corruption of* agile.

Speaking only for myself, that is not true. My position is that each agile practice (or the whole lot together with an appropriate label) makes sense only in certain contexts (say certain types of enterprise software, with certain types of teams) even in the "uncorrupted", "pure" state.

A "pure agile" process is not superior to a "non agile" process de facto.

Agile is not (imo) the "best way we know how to create software". It is *one way we know* how to create software. It has no intrinsic superiority (except against obvious straw men like "pure waterfall" for projects with rapidly canging requirements).

"Post Agile" is just an adjective that describes people who *have used agile extensively*, adopted what made sense, rejected the parts that didn't work (and the hype) and choose to think for themselves. It is not a reaction against the perceived corruption of an originally perfect process.

Just to be clear :-)

Kelly Waters said...

For me, Agile principles and practices provide a framework for people to work in a more versatile way, helping some people/companies to deal with some of the challenges of software development.

Agile is not a silver bullet though, but it is a different way to the more traditional waterfall method and there are pros and cons depending on your situation and preferences.

Expecting any methodology or approach to solve all problems regardless of the skills and experience of those that practice it is simply not realistic, and then there's a danger some people will throw the baby out with the bathwater.

Kelly Waters

Vladimir Levin said...

Thanks for the comments! I'd like to clarify one point. To me the term agile represents neither a process nor a methodology. Agile is a set of general principles that guide software development activity. XP on the other hand is certainly a methodology. Therefore I don't think agile is a "perfect process" but I do think that agile is the best we can do - in the general sense of its emphasis on continuous design, feedback, and results. To my mind there are basically 4 overall ways to develop software - I think of them as "attractors" to borrow a methaphor from chaos theory. They are Phased Development (e.g. Waterfall), Phased Iterative Development (e.g. most RUP implementations), Agile (e.g. XP and Scrum), and Hacking (aka Code And Fix). One can develop working software with any one of these approaches, but to my mind, Agile ought to be the most balanced approach since it is both flexible and disciplined and should have the fewest superfluous documents and the least un-necessary process.

I would certainly be interested in discussion of cases where something besides agile would be preferable.

Ravi said...

Kelly said
"For me, Agile principles and practices provide a framework for people to work in a more versatile way, helping some people/companies to deal with some of the challenges of software development"

The "some" makes all the difference. The "some" (some contexts, some people, some companies, some projects, some practices, some teams) is the essence of the "post agilist" position. But once you adopt "some", you could replace "Agile" with almost anything and the sentence would still hold true.

And thus we are back to Square One.

Hence the "psot agilist " position of "Think for yourself and use what works. don't be taken in by hype."

Vladimir Levin said...

I do think that in general the agile practices are a real improvement over non agile practices, namely ones where communication channels are narrow and document-oriented; large swathes of work are planned far ahead of time; where one sees design documents developed by an architect and handed to programmers who's job is to somehow translate them into code; where there is little room to alter the way things are done as the project proceeds.

I'm not going to make claims about compilers and AI, which Ravi is interested in, nor about other fairly "exotic" domains such as operating systems, graphics, and also resource contrained embedded computing where 8 and 16 bit chips still may be the norm, but certainly for the majority of enterprise software, I would feel confident suggesting an agile approach. That doesn't mean that applying agile *properly* is easy, but I do think that there are benefits to be realized and the effort is worth it.