Saturday, November 26, 2005

Clearing The Fog: Strategy For Spikes to Clean Up Code

I've been working on quite a large project since February. It's hard to believe that it's been just about 10 months already. Such a project, with 20-30 developers, is quite different in nature from the smaller projects I've been involved with before. For one thing, one has less control over the code. People of varying abilities and inclinations are submitting code all the time, so --to borrow Fred Brooks' idiom-- maintaining conceptual integrity is a more difficult proposition. As such it is not unusual to run into less than optimal code in the application. The problem is that while I may sense bad smells in the code, in the midst of all the complexity it can be difficult to ascertain exactly what I should do about it. In some cases, I've found that simply trying out various refactorings just isn't as helpful as one might hope. One strategy I've found helpful is to start a completely new codebase and to quickly write some very simple code to model the functionality in question. That way I am not burdened with the production code base. I write the code test-first in order to generate what I think is a reasonable solution to the problem. Often I can get working tests and a reasonable sense of what the solution I want looks like much more quickly than I could by working on the production code base.I consider this to be a spike, but on our project, most people refer to a spike as an experiment applied to the production code. Once I've found a solution that seems to fit, I try to adapt it to the actual project. Often I can refactor reasonably large parts of the, ummm, less than optimal code, to match the general layout of my new solution to the problem. Sometimes I have to re-write some parts entirely. In any case, I've found that doing this kind of spike, away from the production code entirely, can help to quickly generate a fresh view of the problem and its solution.

I had a look at the definition for a "spike" on the c2 Wiki and came upon concepts that appear basically identical to what I've described in this post:

Wednesday, June 01, 2005

Problem Solving and Avoidance

Today a colleague asked me for my take on a problem he was working on. He was trying to keep track of allocations for a given network of facilities. So for a particular facility, say an oil well, the allocation result would store the path that the oil took through various facilities. A more concrete example: Oil Sales Allocation, from Battery B2 to Well W1: W1 <- B1 <- B2. His approach at the database level was to store each segment of the path is a database record and to connect the top record to an allocation result record. The problem he was running into was how to match up a path to an allocation result. So suppose I know a given allocation was supposed to follow the path W1 <- B1 <- B2, how to I find the matching allocation in a reasonably efficient way. I suggested that he try something simpler to start with: Just store the allocation path as a string inside of an allocation record. That way, it's easy to match up the expected path with the actual path. If something more normalized becomes required at the database level, one can always use some PL/SQL (we're using Oracle) to create the needed string from a more complex database structure. He walked away with some enthusiasm for the idea. Now, we'll see if this particular idea actually works out in practice, but the point is that if you're running into a tough problem, the simplest approach might be to think of ways to avoid having to solve it altogether. It doesn't always work, but I have found this sort of approach to be a useful way of simplifying problems and therefore the corresponding design.

Sunday, March 20, 2005

Rapid Convergence

In agile development, the customers decide which features to implement and when to implement those features. In XP the customers organize this information in the form of story cards. Each story card is assigned a priority and reflects a feature with real value to the business. I've found however that ordering features strictly by business priority may cause thrashing in the design of the code. So, while I still consider business priority to be a very important criterion for selecting what to work on, I also have come to think that technical people should be involved in making these kinds of decisions. In particular, I believe that with some thought, feature development can be organized to encourage rapid convergence of the code. In other words, one should try to avoid a situation in which one starts building a particular piece of functionality which stays in the system for a long time and becomes embedded in the design only to introduce something much later which requires that design to be completely overhauled.

Now, that doesn't mean that one should make the mistake of developing a very general kind of design right from the start, the so called Big Design Up Front (BDUF). However, I think that customers and developers can work together to organize stories to maximize convergence. By convergence, I simply mean that the design of the system accomodates new stories so that their introduction is relatively routine and doesn't require a lot of re-design. As an example (without going into detail), the application I am currently working on handles many different "allocation methods." However, only now are we implementing stories that define more complex topological scenarios, i.e. "This thing which used to connect only to one other thing, now connects to many." From my point of view, this kind of situation represents a example of an opportunity to enhance convergence: Develop the application with just one allocation method, but make sure all of the topological issues are handled early on.

In conclusion, iterative development is the best way to develop an application and BDUF in almost all cases is truly a bad idea. The design should indeed continue to evolve as a software development effort continues. However, by judiciously setting priorities on stories, one can encourage the design to crystallize more quickly.

Saturday, March 05, 2005

The KISS Principle

As software development continues its mad pursuit of new ideas and paradigms, whether it's about Design Patterns, Refactoring, Aspect-Oriented programming, or something else, people's tendency to forget about the basics of good programming continues to be a source of constant frustration for me. Of all of the software development principles and heuristics, the one I consider to be the most fundamental is the KISS principle: Keep It Simple, Stupid!

Good programming means keeping functions small. I've learned to adhere to Christian Pekeler's idea that one should try to limit functions to 6 lines of code. In working with this idea together with TDD, I've been surprised at how strong the effect on the design is. It's a simple idea indeed, but a powerful one!

Another basic principle I value highly is to remove duplication. If adding or modifying functionality has you poking around in many different places there's something wrong. Don't wait for it to get worse. Fix it now. It doesn't make sense to allow things to deteriorate to the point where working with the code becomes painful. It's easy to fix problems of duplication early; fixing them once they become truly painful is hard because by then the code has become inflexible. This wonderfully simple and useful idea leads the design patterns emerge naturally in the code rather than forcing them into existence by force of an architect's will.

So, is there anything wrong with Object-Oriented Programming? Design Patterns? Architectural Patterns? Aspect-Oriented Programming, or what have you? All of these ideas are presented by intelligent people genuinely trying to make a contribution, but all too often developers forget about the KISS principle and the basics of good programming.

Agile Developers are dedicated to keeping the design as appropriate and clean as possible. This is not a haphazard or tentative commitment. Agile Developers do not "clean up" the design every few weeks. Rather, they keep the software a clean, simple, and expressive as they possibly can, every day, every hour, and even every minute. They never say, "We'll go back and fix that later." They never let the rot begin. -- Robert C Martin, Agile Software Development.

Monday, February 21, 2005

Variations on TDD

TDD is the cornerstone of XP. TDD requires one to write tests ahead of code. The basics of TDD are laid out in books by Kent Beck (Test Driven Development By Example) and others. However, as time has gone by and XP has evolved, it appears as though different ideas about how to write code test-first have emerged, all claiming to be TDD. Since I am a strong advocate of XP and TDD, I thought I'd lay out some of these variations I've seen in the wild:

  1. Traditional TDD: Development is done one simple test at a time. Refactoring is applied frequently at a small scale. One starts with an intentionally naive design and allows the addition of new tests to guide the evolving design to greater levels of complexity. One can do some up front thinking about a problem, including brainstorming, using diagrams or crc cards, but the focus is to keep the design as simple as possible with each new test
  2. Proactive TDD: Proactive TDD is similar to traditional TDD in that one uses small-scale tests. However, one generally develops a fairly established design ahead of time and the tests are written to realize that design. This approach is less flexible than traditional TDD, but the feedback cycle is still short and provides the opportunity to re-think the design if it looks as though it is going in the wrong direction.
  3. End-To-End TDD: In this variation, one does not write small-scale tests. Instead one starts off working with what are essentially acceptance tests: High level tests that cover the execution of code for a story or a substantial part of a story. Such tests may use something like the FIT framework or they may involve calling a high level API inside of xUnit. To make one test pass, quite a bit of code must be written through all the layers of the application. The presumed goal of this approach is to avoid having to modify many small tests as the application logic changes. The drawback is that this is not really a good design technique since one cannot apply a short test-code-refactor cycle. Overall I would say this approach discourages frequent refactoring and leads to low quality code. If one does decide use this approach, In my opinion one should be prepared to do a lot of careful design up front.
  4. Mock-Driven Development: I am unfamiliar with this approach, but there appears to be a school of developers who produce even more granular tests with the extensive use of mock objects. The sense I get is that this kind of development is perhaps more similar to Design By Contract and ties tests very closely to the implementation details of an API. So far I can't evaluate this method, but I have to admit that I am biased in favour of testing state before and after a test rather than worrying about the actual calls made.
Based on my experience thus far, I consider Traditional TDD to be the best approach in most cases and Proactive TDD to be acceptable in some cases. I believe that all in all traditional TDD minimizes the risk of ending up with unmanageable code. While I am a strong supporter of acceptance tests, I believe that all application code should be developed using a short test-code-refactor cycle, and therefore one should use traditional TDD techniques to write the code that will eventually make the acceptance test pass. Without the traditional TDD, one loses the ability to frequently refactor the code in small chunks and to get a good sense of its overall effect on the design.

Monday, February 14, 2005

Appropriate Metaphor For Software Engineering

Jeanette Winterson is one of my favourite writers, especially her novel The Passion. This is from the monthly column she writes for her Web site:
My friend Mona Howard, who is 84, told me she had been to see the film Touching The Void, about the mountaineers, one of whom makes an incredible survival journey with a broken leg. He got through it by setting himself only what he could achieve without agony giving way to despair. So he would hop for twenty minutes, then rest, then hop again.

Saturday, February 12, 2005

What makes a successful XP team?

I've recently had the experience of working with a productive team and I thought I'd share the characteristics that I believe were important to its success at delivering on time with a low defect rate:
  • Avoid politics: Everyone on the team was always thinking of ways to improve the quality of the product and the code. There was no sense that people were motivated because they were trying to impress a boss.

  • Do not over-engineer: The focus was on getting results, not on stroking one's ego or on doing software engineering for its own sake. All too often in the OO world, I see over-engineered code that seems to be developed out of some desire to be clever or to copy something from a book.

  • Remain critical: There was a strong desire to root out problems in the design. In other words developers did not fall in love with their designs, regarded the code critically, and were eager to make changes to improve it.

  • Allow some slack: Developers were able to negotiate periods of time to focus on cleaning up the design of the application rather than pushing forward with user stories. Sometimes it is necessary to go in and thoroughly clean up, especially when dealing with a legacy codebase.

  • Maintain discipline: Discipline and focus was very good. Planning was done in 2 week sprints and people were not getting constantly interrupted during a sprint. One could count on the estimates provided for stories.

  • Have a separate QA group: A separate QA group producing automated end-to-end tests was very useful. Together, the end-to-end tests produced by QA and the more granular tests produced by the developers provided a high degree of confidence in the code and allowed fairly significant refactorings to be done.

  • Promote a team culture: Developers maintained high morale by taking an easy-going approach to the project, even at times of high pressure. On our team, the developers and QA people played video games at lunch and during the last half-hour of the day. Any kind of activity along these lines promotes a sense of team-identity. Whatever it might be, it is a good idea to come up with something non-work related the team can do together on a regular basis.