For many IT managers today, the task of managing software projects and development teams is often a game they feel they can't win. Software development is increasingly complex, and keeping team members focused is challenging. Factor in never-ending feature creep, and adhering to schedules is next to impossible.

Is there anything that can help? Many managers and developers swear by agile programming and project management as the answer to some, if not all, of the challenges. This article examines agile programming and project management. Before we get into the details of agile development, though, let's get to know a new friend. His name is Neil.

Meet Neil

Neil and his software development team are charged with creating a new customer sales reporting module for the marketing and sales teams. After a couple of meetings with marketing and sales, Neil and team spend several days working on a project specification document. They hold two additional final meetings with the marketing and sales departments to finalize plans. The marketing and sales teams accept the project specs, and Neil's programming team gets the green light to proceed (all told, the meetings chew up 10 days).

Neil and his team have a progress meeting each Monday, but realistically the most tangible thing anyone gets from these meetings is a free doughnut or two. When he isn't himself writing code for the project, Neil spends whatever spare time he has with the two groups of coders that comprise his team. Rather than actually managing much, however, Neil spends most of his time with the groups' code.

The team works on the project for more than five weeks. The project is due this week. However, with testing at least another week away, delivery looks to be in about three more weeks. If Neil's lucky, he'll deliver the project in eight weeks instead of the projected five weeks. And that's if

• no one else changes the specifications or asks for any more features; the marketing team has already demanded two additional features that weren't originally scheduled, and now the sales team wants the ability to display sales history for more than one customer at once
• Bob and his group stay focused on the remaining deliverables; Bob leads one of Neil's two programming teams, and when focused, Bob's group does great work — but too often, Bob and his group veer into the weeds chasing a feature or capability that isn't directly in line with what needs to be delivered
• Neil can find the time to spend with Clair and her group, as originally planned; this group is just coming up to speed with web services and needs some direct help from Neil to ensure that its part of the project works as expected — one of Bob's coders, Katherine, could be a big help with this part of the project (without affecting Bob's group's overall productivity much), but Clair's and Bob's teams see themselves as separate entities and don't work well together
• no more unexpected problems are encountered; some performance problems occurred with the original database design, and fixing them cost several days
• Neil can synchronize testers with the code he has yet to deliver; they are busy with other projects and expected by now to have the code to test
• the documentation team is up to speed with the latest added features; the last that Neil heard, the documentation team had just finished explaining the customer sales history component, and that is in for big changes before the product delivers

And to think that Neil once thought he'd landed his dream job! This important project is going to be at least three weeks late, and despite the dramatic impact of outside pressures, the blame for the project's tardiness will fall on Neil's shoulders. What could Neil have done differently?

Enter Agile

Agile software development (ASD) is one of today's hottest software development concepts. ASD isn't a formal methodology but rather a conceptual framework that floats above many specific agile development and project management methodologies. For example, Extreme Programming (see "Zen and the Art of Extreme Programming," March 2005, article ID 19882 at SystemiNetwork.com) is a formal methodology for implementing ASD among programming teams. Scrum is a project management methodology that embraces agile concepts.

ASD is defined by four key tenets, known collectively as the Agile Manifesto. The Agile Manifesto (agilemanifesto.org) says:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Let's review the detail of each concept a bit. If you're cynical like me, you'll quickly recognize that much of what's at the core of the Agile Manifesto is simply common sense and many old concepts with new names. The last thing the software development community needs is more new terms to describe previously existing concepts and ideas. Having said that, however, I advise that you not skip immediately to judgments about agile software development. ASD has a lot to offer. Read on — with an open mind.

Value individuals and interactions over processes and tools. It might seem odd that this concept says "individuals" instead of "teams." However, "team" is implied when the manifesto stresses individuals and interactions. It's very important that your team members have a holistic sense of their work, that they recognize that they are a team and are working toward common goals. Team players should be able to move easily from programming group to programming group without raising the "it's not my job" flag. If you manage the individuals and their interactions, the result is a team! (Neil hasn't managed the interactions of his team members very well. Had he managed Katherine a little better, she could have helped Clair's team with the web services.)

A highly skilled, veteran carpenter can build a better cabinet with a handsaw and a hammer than a rookie carpenter can with a table saw and a nail gun. So it is with software. Tools are indeed important, but without the core skills and abilities, a debugger with edit-and-continue capabilities won't intrinsically help your team develop better software. That isn't to say that tools are unimportant. But don't focus prematurely on the tools. Build the team, and then let the team select and implement the tools.

Processes are also important — you need to know what's supposed to happen and when it's supposed to happen. A common theme throughout agile concepts, though, is that deliverables count more than planning for deliverables. (Neil made a mistake by burning several days creating a master plan rather than focusing the team on necessary, early deliverables.)

Value working software over comprehensive documentation. Everyone knows that documentation is necessary. But ASD encourages short, rational documentation. ASD proponents argue that overly long documentation takes too long to create and too long to maintain. As these large documents get out of date, either they are ignored or they mislead and derail those who read them.

In the absence of lots of documentation, ASD suggests that the way you pass knowledge on to new programmers is by having existing programmers share what they know with the new programmers (remember, interaction is mentioned in the first tenet). Robert C. Martin (objectmentor.com), a major ASD proponent who has written much about the subject, suggests this as Martin's First Law of Documentation: Produce only those documents for which you have a significant and immediate need.

In our example, the documentation team probably wasted quite a bit of time in documenting the project to too fine a level of detail before it was even finished. The documentation team should have also been working with Neil's team as an integrated part of the development process rather than being outside of the development process.

Value customer collaboration over contract negotiation. Software is an intangible commodity, and as such, it's a challenging product to deliver for a fixed price on a fixed schedule. ASD proponents argue that any software project treated in such a fashion is doomed to failure (see "A Monumental Software Development Failure," below). Some software managers assign tasks based on the latest Gantt chart, and then they take the path of least resistance, assuming that those tasks will be completed as expected, when expected (Neil did this). In the realm of software development, perfect adherence to a fixed price and schedule rarely happens in real life.

Successful projects require good, constant communication between not just the manager and the team but also with the customer and/or the users of the new software. The point here is to identify problems and concerns earlier in the process rather than later. Implicit in this point is that as you discover these problems and concerns, you can keep your customer's and/or users' expectations in line with the reality of the project (something Neil clearly didn't do by simply accepting change orders without explaining the ramifications of these changes).

Value responding to change over following a plan. This tenet grows out of the last one. Rather than locking down a contract and pushing your way through with your head down — as though nothing is going to change (and you know it will), ASD not only accepts, but encourages, change. The more nimble your team, the more you can react quickly to changes, either those imposed internally, such as Neil's database problem, or those imposed externally, such as the marketing and sales teams' out-of-scope requests.

Other Agile Concepts

In addition to the four primary ASD tenets that the Agile Manifesto declares, an agile process has several crucial components:

Work in short iterations. ASD requires that its processes all be highly iterative. Rather than spending lots of time creating a master plan and then moving linearly through each phase (i.e., analyzing, designing, coding, testing), the team repeats each phase on a much smaller scale within iterations.

Use timeboxing. ASD iterations are governed by timeboxing. That is, each iteration has a hard-and-fast schedule for the deliverable (usually somewhere between two and six weeks). This deliverable though isn't the completed project but is simply a part of the completed project. Timeboxed iteration due dates must be met, even if that requires limiting what gets delivered. The timeboxing concept might seem at odds with the tenet of "responding to change over following a plan." However, the timebox is an ASD concession to the real world; it's timeboxing that ensures that you deliver something on a reliable and recurring basis.

Deliver continually. For each iteration, the team needs to deliver a coded, tested part of the software. Not every iteration's deliverable gets promoted to production, but it could. It's important to understand that the iteration might not deliver everything it is supposed to. In Neil's case, had he been using timeboxing and continual delivery, his database problem wouldn't have kept the team from delivering something for that iteration.

Inspect and adapt. Things are going to change. At the end of each iteration (as governed by the timebox), the team evaluates and changes the plan if necessary. The customer and/or the users and all the team members are apprised of any changes. At the end of each iteration, the next iteration is planned. Planning in iterations limits your "planning horizon" to what you can see (again, somewhere between two and six weeks out).

For more information about ASD, its theories, and its related methodologies, see Find Out More, below.

Neil Gets Agile

If Neil were to use an agile process, he still might not deliver his project on time. However, he probably would do a better job of managing the project and the expectations about the project. Agile processes could affect Neil and his team in these ways:

Lightweight planning up front. Neil and his team, as well as users (in his case, users from the marketing and sales departments), would spec out the project quickly and with just one or two meetings. They would develop a general overall plan, but then that plan would be quickly broken down into (in Neil's case) two-week increments.

Iterations and continual delivery. The team would deliver some working software every two weeks. Even though that deliverable might not be integrated into the final, full product every two weeks, concrete progress to move the project forward must be made every two weeks. At the end of each timeboxed iteration, Neil, his team, and the marketing and sales representatives would review what's been done and what needs to be done, as well as what changes might be needed. Because users would see software much earlier in the process, they could more accurately express what they need the software to do differently. Also, because the sales and marketing people would help plan the next iteration, they would be kept apprised of the overall schedule and have more reasonable and real expectations about delivery dates. The testing and documentation teams also would check in at the end of each iteration and, if necessary, integrate the just-delivered work into their schedules.

Lots of team interaction. Neil would no longer have his Monday-morning doughnut meetings with his programming team. Instead, they would meet every working day for a very short stand-up meeting (i.e., no longer than 10 minutes). They would discuss the previous day's progress and identify hot spots and trouble areas and then make the plan for the rest of the day. At all times during these stand-up meetings, the team would focus only on what is necessary to provide the deliverable for the current iteration. Bob and his team would be kept reined in because the overall team's purpose and goals would be shared daily. And, because the team has an overall plan and hard-and-fast deliverables, Bob's and Clair's teams would feel less like two separate teams, so Katherine should be happy to work with Clair's team to help with the web services.

Agile Isn't All Rainbows and Puppies

As you read through the numerous books and websites about agile software development, you're exposed to many interesting concepts and ideas. But if you're like me, you can't shake this vague feeling that something isn't quite right. ASD proponents have a cultlike fog about them. You can't help but get the feeling that "Yes, there is something to this, but something smells a bit fishy just the same." For an old-timer like me, some parts of ASD feel just a little too idealistic and unrealistic.

ASD dispatches or minimizes some fundamental concerns. ASD encourages very little up-front design. It can seem like ASD is saying, "Just start coding; you can define the specs around the code you write." ASD favors customer collaboration over customer contracts. This approach might work well for internal software, but for the contracts that I've been involved in, it would not have worked to say, "Hey, let's forget about what you want and when you want it. Instead, let's have a rap session every two weeks. If you like what you see, you pay me. If not, I'll work for two more weeks and we'll see if you like that. Our contract doesn't need to mention payment. Collaborating with you is its own reward."

Having raised an eyebrow to ASD's absolute definition, know that ASD offers your team, especially when used with prudence and discipline, a strong foundation on which to tailor a productive methodology. Agile concepts and principles have many positive and valuable aspects. Use the techniques where they work for your team; modify them when they don't. For many a System i team today, the adoption of any methodology is sorely lacking. Those teams could do a lot worse than to start with ASD.

On the Path to Sanity

One of ASD's strong points is that it acknowledges that software development is a living, breathing process that can't be governed by the commodity-centric processes. Although none of the agile mantras directly mention it, one of agile's undertones is simple: discipline. And, agile or not, that's what it all comes down to. Do you have the discipline to make a plan and then work that plan? Agile development isn't a silver bullet for acquiring that discipline. Rather, ASD provides the framework for acquiring and building that discipline.

As you ponder ASD for your team, remember that change, any change, is often viewed as a negative. Introduce ASD to your programming organization with political tact and care. Its success depends on the entire team believing in it. In a software development world gone mad, you need all the help you can get. Adopting agile software development principles won't provide a cure-all, but the core values and concepts in ASD can set you on the path to sanity.

Roger Pence is education director for ASNA, an application development tools company, where he focuses on helping customers build browser- and Internet-based line-of-business System i applications. Roger has spent many years writing and speaking about the System i.