How to Produce Good Software, Part 8

In part 7, I listed the main aspects of product develoment that I intend to write about. In this part, I talk about iterative development and how it affects the way I write software.

As I mentioned in part 6, I try to get beta 1 of a product out within about 6 months. I'm a big fan of iterative development, and like to break down a particular release into smaller iterations that are between 1 and 2 weeks in duration.

Ideally, each iteration is an installable product that includes binaries, examples, API docs and a user guide, even though in early iterations these are admittedly sparse. There are several benefits of this approach. First, the result of an iteration can easily be tried by anyone, including people who might be part of an early access program. Second, the total end-user experience gets incrementally refined along with all other deliverables, rather than just before the release. The installation experience, user guide layout, configuration instructions, examples and other such artifacts are all amenable to improvement over time, so the sooner you start getting feedback on these aspects, the better.

When I'm breaking a single release into a series of small iterations, I use a few rules of thumb:

• focus on getting the basics working first
• implement useful subsets to keep the iterations small
• drop features from iterations if they threaten the timeline
• features are continually rescheduled, so don't sweat it
• select features that keep the process enjoyable

The last point is particularly important. Developing software is more like a marathon than a sprint, so pacing the team is critical to avoid burnout. I often ask my team which features they would be most interested in implementing in order to keep iterations attractive and fun to work on.

Here's an example of how I selected features for the iterations leading up to Glue beta 1. The three core features of the Glue web services platform are binding to a service, invoking a service, and publishing a service. In the production release, these features would clearly have to include security, performance optimizations and other such enterprise factors. But in the early iterations, I just wanted to get each feature working in a minimal form to get a feel for the APIs and the user experience. So in iteration 1, I implemented each core feature in its simplest form, with support for primitive arguments only. To support this goal, I had to implement a tiny subset of an HTTP server and HTTP client, which was fairly straightforward, as well as a generator and parser for the small subset of WSDL that supports messages with primitive arguments. During the development of iteration 1, I also created some simple examples and a user guide with early chapters for installation, configuration, overview and running the examples. At the end of iteration 1, I created javadocs and packaged up the whole thing using the ZeroG InstallAnywhere utility.

In general, I write each example to showcase a single feature and avoid pulling in other extraneous APIs. It's frustrating to me when I read examples that require knowledge of several other irrelevant features. That being said, I like to include examples towards the end of a user guide that show how features are typically used together when building a real-life application. Iteration 1 had only a few examples, each 3-4 lines long, which demonstrated each core feature.

Once iteration 1 is finished, I have a better idea for which areas are easier than anticipated and which might require deeper thought. For example, during Glue iteration 1 I discovered that HTTP has quite a complex specification, and decided to spread the implementation of its various features across several subsequent iterations based on their relative importance. For example, I decided that support for HTTP chunking was not that important and put it off to a fairly late iteration. In addition, implementing all of HTTP in a single iteration would have been really boring and I wanted to keep each iteration interesting.

Once Glue iteration 1 was completed, I started on iteration 2, adding features such as support for complex types without inheritance or polymorphism and support for HTTPS. In iteration 3, I added support for inheritance in complex types as well as support for arrays. Hopefully you get the general idea: each iteration introduces a couple of new features as well as fleshing out some of the previous features.

I've found that what most developers find the hardest about iterative development is accepting that early designs are immature and will almost certainly be improved and generalized. A lot of people start with an initial design and then almost immediately try to improve it in anticipation of features that are scheduled for a future iteration. This in turn chews up more time, delays the iteration schedule, and often results in a design that either misses the mark or is overengineered. The Zen of iterative design is that you improve design during a particular iteration only when necessary to support a feature in the same iteration. Another way to describe this is "on-demand design".

As you change the design during each iteration, there are many times when the code needs to be refactored. In the old days, I would refactor using various utility scripts, such as SED (the UNIX stream editor). Nowadays, IDEs with refactoring make this far easier. I'm a big fan of aggressive and continuous code improvement, and usually spent an hour each evening reviewing the entire Glue code base for opportunities to improve the design and/or implementation.

In the next part, I'll discuss API design.

Comments

[Paul Holser]

good stuff, graham. curious what steps you take during your refactorings/redesigns to defend against regression -- to ensure that what worked before still worked after the changes. i've found the xUnit frameworks valuable for this purpose.

[Graham Glass]

Hi Paul,

Good question! I'm going to cover that topic when I talk about "testing".

Cheers, Graham