Saturday, March 31, 2007

Renovation Cost Overruns

I was listening to CBC radio today and they were talking about renovations. It turns out that the average cost of a renovation in Canada is about $5000 and the average cost overrun is 88%! I don't know about other people out there, but I tend to be somewhat ashamed of the poor track record software projects have - they always seem to be late and over budget and more buggy than one would like. It's kind of comforting to note that ours is not the only industry where such things happen. If I were hiring someone to do a renovation, I'd definitely implement the plan as a Scrum project, getting fresh estimates on a regular basis and tracking progress against the original estimate. I'd also have multiple "releases" i.e. I'd make sure one job was competed in its entirety before I agreed to go ahead with the next one.

Monday, March 26, 2007

Costing A Project Using Scrum and XP

At the last EAMUG meeting Mike Boroditsky came up with some good questions about doing an initial costing of a project. Let's say a potential customer comes along and asks you how much she would have to pay for a piece of software she has in mind. How does one go about providing such an estimate in an agile context? If agile software is about iterations and feedback, is there any way to give a customer an up-font estimate? The idea in XP is that every project balances 4 principal variables: Time, Cost, Scope, and Quality. XP fixes the time, cost, and quality. Therefore only scope can be varied during the project. Based on an initial estimate, the developer will commit to a particular schedule, but both the developer and customer must be flexible with respect to scope - the nature of the work that ends up being done. See here for a nice elaboration. I think the reality in agile projects is that the time dimension can also stretch as long as the customer is involved in managing the schedule.

Here's how it might work in context of Scrum/XP: The project starts up with one (or perhaps several) initial meetings. In these meetings the developer and the customer get together and build up a product backlog. The product backlog is essentially a list of features sorted in order of descending priority. During this process, the developer and customer work together to establish a shared understanding of what the customer is trying to accomplish. Once an initial product backlog has been created, the developer goes over each item in the backlog and provides an estimate. Since each item in the product backlog may expand into multiple stories when it's tackled as part of a sprint, such estimates as not as reliable as etimates for stories. Nevertheless, one should be able to generate reasonable estimates based on how long similar features took on past projects and on a sense of the complexity of given items. Once these estimates are added up, one can tally them up and provide a total time estimate for the
customer. These estimates will be tested against reality in the very first iteration; the burndown chart will make progress clearly visible to all. If progress is totally off-pace from the initial estimate, then that triggers an immediate signal that something is wrong.

How about fudge factors? I've heard people say they expect some estimates to be too high and others to be too low so that the average should work out to be about right. My experience suggests that estimates are far more likely to be too low than too high. Therefore it may be a good idea to apply a fudge factor to the project. In an agile context where everything is out in the open, it would be important to share this information with the customer. The developer may say to the customer, "the estimates in the product backlog suggest this project will take 3 months to develop but it's very easy for such estimates to be off by as much as a factor of 2. Let's budget for up to 6 months." Note however that the estimates are not being inflated - the budget is being adjusted as a matter of risk management but the estimates are not being changed at all.

Watching the burndown chart (which is a visual representation of velocity), one can quickly develop a sense of whether the project is on track. After each iteration, the customer can re-prioritize, add, and remove items. All changes are put back into the product backlog and the developer re-estimates all the items after every iteration. The "red line" allows the customer to clearly see what low priority items are falling out of scope. In an agile environment, the customer is not expected to be able to come up with perfect requirements up front. The customer is allowed to learn more about what she wants as the project goes on and as she is able to interact with the actual working application. The developer is also not expected to understand the requirements perfectly. It is assumed that the developer will misunderstand some requirements such that the application will need to be refined. It's a process of give and take - but again, the overall success of each iteration is assessed by the customer.

What about quality of estimates? The more familiar a developer is with a given type of requirement - the domain as well as the technology - and with a particular client, the better the estimate. If you've worked with a client before on a similar project, your estimates are going to be much more accurate. The higher the estimate, the more likely it is to be innacurate. If you think something will take 1 day, it may take a bit longer. If you think something will take a month, it becomes difficult to understand where that number is coming from. Therefore I suggest that the largest estimate for any given item on a product backlog be one week. If you think something will take more than a week, then you should find a way to break that requirement up into smaller pieces. It's important to realize anything that will require research is highly uncertain and should be approached with care.

I think the value of this approach is that it lets the customer see progress very clearly. After every iteration, the customer gets new features as part of a working application. Also, if the customer is unhappy with the progress being made, every iteration is an opportunity to change things, or in the extreme case to cancel the project before it becomes too expensive. Importantly. the developer does have the tools to provide an up-front estimate of time and money for a project.

EAMUG

I've started an agile group here in Edmonton, EAMUG - or Edmonton Agile Methods Users Group! So far I've met some good people and I presented at our first meeting, which I think went pretty well. Rod Potter, who works at IBM, made a fantastic room available for us at their innovation centre. I hope we'll have more good meetings in the future!

Thursday, March 22, 2007

Orcs And Elves

From John Carmack's blog:
O&E development was started high-end first with the low-end versions done afterwards. I should have known better (Anna was certainly suspicious), because it is always easier to add flashy features without introducing any negatives than it is to chop things out without damaging the core value of a game.
I think this is true of all development, not just games, and it's why adding a lot of flashy stuff up-front is a bad idea. Unfortunately, users often are much more impressed with the bells and whistles than with core functionality. Striking the right balance is one of the hardest things about software development. Agile approaches emphasize a vertical style of development - get it working end to end, then add more advanced capabilities, but it's often hard to do that with users often demanding cosmetic features that are not necessarily easy to implement and impose a debt load for ongoing development.

Wednesday, March 07, 2007

What Is a Successful Project?

What Is a Successful Project?

A few years ago I would have promptly answered that a successful project is on time, on budget, and does what the customer intended it to do. Over the course of the last few years though, I've learned that things aren't quite so simple. Developing software is a human activity and it's taken me a while to figure out that 'success' is a profoundly subjective notion.

Let's say youre a painter and you've been hired to paint a house. You go in, do the job, and when the customers look at what you've done, all you hear are complaints: They can see a few rivulets of paint near the ceiling, the paint job is not perfectly even everywhere, etc... The fact is that there's no such thing a s a perfect paint job so it's up to the customer and the painter to come to an understanding of what's reasonable and what's not. Trying to settle the matter confrontationally is unpleasant for both sides and should only be a measure of last resort.

The same is true in software, only more so. The degree of sophistication to which any given requirement can be implemented can vary enormously and it's important to come to a general understanding with a customer about what is expected and how much time and money will be involved. As a simple example, let's consider Web form validation. A simple way to validate data entered on a Web site is to have the application issue error messages that are displayed when a form is submitted. However, in addition to this basic type of validation, the customer may want validation to occur 'live' as the user is filling in fields on a form - in practice this rquires either client-side validation with Javascript or asynchronous server calls using 'ajax'. The customer may also want automatic filtering on fields to prevent the user from entering incorrect data - a typical case in point is filtering provinces/states once a country has been selected, and then additionally filtering cities once a province/state has been selected. Depending on what the customer really wants, a story may simply state the validation rules for a given form or it may describe these kinds of things in detail. In the first case, XP would indicate the simplest approach applies and the customer would have to draft a subsequent story to enhance validation. In the second case, fancy validation would have to be considered a motherhood story - something that is an essential part of the application right from the start and cuts across all stories (other examples of typical motherhood stories include Security, Performance, Failover, etc.). Thus, the Fancy Validation motherhood story would be an aspect of development for every form and support for it would have to be built into the code framework fairly early on.

Every development activity can be approached horizonally or vertically. Implementing fancy validation right away across the board would be an example of a horizontal approach. The vertical approach would suggest getting substantial parts of the application working satisfactorily before contemplating such niceties. Agile development tends to emphasize a vertical (get it working end-to-end first) approach, but it's essential to pay attention to the customer. Perhaps the customer can be convinced to alter priorities; on the other hand, the customer may know exactly what he or she wants and having raised the matter and given appropriate feedback with respect to risk and cost, etc, it would be the developer's job to carry out the customer's instructions.

In the previous paragraph I was trying to describe the kind of situation where the developer may think that everything is ok but the customer isn't happy. It's even possible the customer is being unreasonable but the bottom line is that a project where the customer is not happy cannot be a success, even if it appears to meet the stated requirements and is also on time and on budget. Now let's suppose the developer and the customer meet to discuss a new project. The developer provides an estimate of 6 months. 2 months into the project, the developer shows the customer the burndown chart which clearly indicates, given the progress made so far, that the project is much more likely to take about a year. There could be many reasons for this to happen. Perhaps as per the earlier discussion, the customer wants features to be implemented in a more sophisticated way than anticipated. In any case, the customer decides that despite the delay, the project is worth continuing. At the end of a year the project is delivered to, and accepted by, the customer. On one hand this project has exceeded its initial estimate by 100%! On the other hand the customer made the decision to continue and is satisfied with the result. It's probably reasonable to suggest that this project is a success, in spite of the budget overrun.

While being on time and on budget is important, and it's also important to satisfy the stated requirements for a project, the reality of software development is that constant communication is essential. Ultimately only the customer and the developer can together decide whether a project should be deemed a success or a failure. There doesn't exist any infallible objective measure of success. If you ignore customer expectations, you may end up delivering something on time and on budget that the users aren't happy with, and then you'll spend a huge amount of time arguing over whose fault it all was - was development not getting the requirements right or was the customer not being specific enough: The kind of confrtonational quagmire one often runs into in waterfall-ish projects. Agile software development provides the tools to resolve these problems, but both the customer and developer sides need to put in the work to strike the right balance between "lots of cool bells and whistles" and "a lean, focused, no-frills piece of software that does what it's supposed to do." It's also a good idea to always think about the difference between the horizontal and vertical styles of development. Some features are best implemented horizontally and others vertically. In an agile development context such decisions ought to considered frequently and reviewed often. I hope this helps you to make your next project a success - whatever that means!