What is Agile Software Development, Really?

"Agile" is one of those ubiquitous buzzwords in the software industry. It is also one of the most misunderstood.

The agile way of working is a very different experience to traditional procurement processes and project management methodologies. Lots of people and companies say they "do agile". In reality, few do it properly.

So what is agile software development, really?

The waterfall method

Agile software development cannot be properly understood without an appreciation for how it emerged out of a rejection of conventional project management processes.

Traditionally, the process of making software was a straightforward linear process that moved through a series of distinct steps, starting with the gathering of requirements, moving on to planning and scheduling, followed by a lengthy process of product development, and ending with a period of testing that precipitates the roll-out of a complete, finished product.

In the software industry, this traditional approach to project management is known as the waterfall method. The analogy is that the roll-out of a software product is a bit like a waterfall: a comprehensive product is released in one go towards the end of a long period of design and development.

Waterfall workflow

For a long time this was the conventional means of making software. But as software became ever more complex, limitations in the waterfall method became apparent.

By the 1980s and 1990s, traditional product development practices started to be rejected by many of the world's top software engineers.

There are three main problems.

1. Software engineering is a design process

The first problem with the traditional approach to software engineering is that it treats the process of making software as something entirely predictable, that software can be engineered to a detailed blueprint and specification. It relies on some kind of prophet being able to devise the perfect roadmap that predicts exactly what will be delivered week by week.

In reality, software is extremely difficult to plan so precisely.

Why is this?

Well, a great deal of the process of software engineering is more about design than engineering.

A structured, engineering-based approach only gets you so far. Great software is made when a project timetable gives flexibility to iterate aspects of the design, to try out different approaches to problems, until you're finally happy with the result. As with any design process, by iterating back over the design, it becomes more refined.

And you can't foretell everything. There are just too many little details and nuances in the source code of computer programs. Even simple programs are the outcome of thousands of micro decisions made during the course of their manufacture. Each bit of code is never a perfect solution but rather a compromise between a series of competing factors: between security and speed; between functionality and ease of use; between maintainability and the initial cost of development; between future-proofing and support for legacy systems; and so on.

Realistically, many of these myriad micro decisions must be made during the development phase.

The more ambitious the project, the more difficult it is to write an accurate specification at the start. An overdependence on engineering principles is the primary reason why big software projects are notorious for being delivered late and over budget, and for otherwise failing to meet the expectations of the customer.

2. It leaves testing to last

Another big limitation of traditional product development methodologies is that they leave testing of the product to the end. Only at the end of the process will you know if the product you've been building for so long is any good or not.

The moment when you start testing a product and putting it in front of real customers is the most critical moment in a product's lifecycle. The sooner you can get feedback and incorporate that feedback into the ongoing design of the product, the more likely the product will be successful.

This is even truer for software than it is for physical goods. The intangible nature of software makes it difficult to set quantifiable benchmarks for quality, against which the software will be evaluated on completion.

The most practical way to evaluate the quality of a software program is to get qualitative feedback from real users.

3. Software is never finished

The final big problem with the waterfall approach is that it treats software as a static, manufactured product; something that can be designed, built and brought to market, and not changed again for many years.

Software is not like this. It needs to continuously respond to changes in the world around it; to evolve according to changes in the requirements of its users and the technological constraints of the wider IT ecosystems on which the software is run.

Software is never finished. There is always something more to do.

This is particularly true of software served over the web, the most rapidly changing IT ecosystem of them all.

Software re-imagined

By the late 1990s software engineers were collaborating together to devise alternative ways to make software.

Three things happened.

First, a lighter, simpler type of software product was advocated. Software should do less, but do it better.

Second, software engineers experimented with bypassing the whole planning and prototyping stage. Instead, they designed software as they went along. Initial requirements were high level only; the finer details were worked out during the development phase and allowed to evolve over time as the product itself incrementally took shape. Project management processes were rationalised with more responsibility for decision making delegated to small, cross-discipline, self-organising teams. More time and effort was dedicated to producing real, working code.

Third, software began to be developed in shorter cycles. Traditional project management processes remained in place, but rather than being applied to the whole thing, the process was applied to individual components of the product. One small bit would be designed, made, tested, and integrated into the product. Then the process started over with the next bit. Each iteration was treated as a little mini project in its own right.

Waterfall workflow

This new approach made software development much more organic. Software applications started out small but were rapidly iterated. Programs were always in a "working" state but never "finished".

This process allowed testing to be conducted at regular intervals and feedback fed into the product roadmap. Every stakeholder gets to see progress and test the product throughout the whole development lifecycle.

This new approach provided the flexibility to change direction: to change priorities, to respond quickly to competition, and to rapidly patch newly-found bugs.

In its most extreme form, this process has such small iterations that roll-out of software updates is almost continuous. When web-based applications started to emerge in the early 2000s, they were not only manufactured in small iterations, but also released in small iterations. The new lightweight development methodologies are particularly well suited to the web, where technologies and the habits of users are in a continuous state of flux and where software updates can be instantly rolled out to 100% of the customer base. On the web, version numbers are irrelevant.

Agile principles

The introduction of lightweight project management techniques, iterative and incremental development practices, and rapid product release cycles, was a new paradigm in the software industry.

Things came to a head in 2001 when 17 of the world's most respected software engineers met in Utah, USA, and wrote the Manifesto for Agile Software Development.

The new approach to software engineering has been known as agile development ever since.

Agile has a lot of crossover with "lean" and "incremental" product development philosophies, and the "rapid application development" techniques from the 1980s and 1990s. All of these concepts are founded on the same basic principles; they only vary a little in their emphasis.

There's no definitive list of principles that define agile software development. But the following principles are commonly found in literature about the agile software development.

  • Initially, concentrate on things that add the greatest value to your product.
  • Capture requirements at a high level. Keep specifications lightweight and visual.
  • Accommodate change. Allow your product to evolve in unexpected ways.
  • Release early, release often. Deliver working software quickly and frequently.
  • Get a minimum viable product out to customers as soon as possible.
  • Iterate from that baseline. Keep the gaps between releases as short as possible.
  • Roll-out minor fixes and changes rapidly, rather than pushing major updates infrequently.
  • Use customer feedback to direct the future development of your product.
  • Incorporate testing into the production cycle. Test early and often.
  • Measure progress in terms of the delivery of working, testable software.
  • Focus on quality, not quantity of features.
  • Get all stakeholders in a project to collaborate on a daily basis.
  • Prefer face-to-face communication among team members.
  • Have all disciplines work together: design, programming, marketing and management.
  • Delegate responsibility for implementation to small, self-organising teams.
  • Empower teams to make decisions. Allow technical people to solve technical problems.
  • Emphasise good design and quality code. This enhances longevity and agility.
  • Keep things simple.
  • Do one thing at a time. Complete each feature before moving on to the next.
  • Re-purpose as much existing code as you can.
  • Reduce overheads and streamline workflows.
  • Avoid things like prototyping and documentation that do not directly make working code.
  • Work in short, intensive sprints.
  • Keep your team off the phone, email and Facebook during production sprints.
  • Allow time for reflection between sprints. Discuss how things can be improved.
  • Keep internal meetings short, focused and informal.
  • Don't work to rigid deadlines. Quality products take as long as they take to make.
  • Your top priority is to satisfy your customers.

Agile methodologies

There are lots of formal agile methodologies and frameworks to choose from. Particularly popular today is Scrum. There's also Kanban, Extreme Programming (XP), DSDM, Agile Data, Rational Unified Process (RUP), Disciplined Agile Delivery (DAD), Scaled Agile Framework (SAFe), and many others.

All of these methodologies advocate lightweight project management techniques, iterative and incremental development practices, and rapid product release cycles. But they vary in their points of emphasis and the specific techniques and processes that they apply, things like pair programming, stand-up meetings, sprints, cross-functional teams, test-driven development, timeboxing, story-driven modelling, retrospectives, and continuous integration.

Many developers today dedicate themselves to mastering a particular agile technique. Having "Certified Scrum Master" on your CV can certainly pay dividends.

But even more useful than mastering one particular agile methodology is to have a general appreciation of the broad principles of agile product development and a high level understanding of lots of different agile methodologies. Ultimately, no single agile framework holds all of the answers for every organisation and every team and every project at every stage of its lifecycle.

Commonly, teams adopt a mixture of techniques from different methodologies. For example, a team might use Scrum's approach to planning and estimating sprints, and also borrow the idea of pair programming from XP. Techniques and processes from different methodologies are broadly complementary. The best approach to agile is to cherry-pick the methods that you think will work well for you and your team, and to tweak your approach on a project-by-project basis, and over time as a codebase moves through its lifecycle.

Agile is all about adopting a flexible attitude to product development. The idea of steadfast adherence to one particular structured approach to agile rather contradicts that ideal.

Conclusion

Agile is no panacea to the software crisis.

Agile processes certainly work brilliantly when it comes to building big, complex, unpredictable systems. But small and simple and predictable projects – such as the development of corporate "brochureware" websites and even online shops, which tend to follow common models – often turn out pretty well under standard procurement processes.

"Pure" agile tends to suit teams made up largely of experienced software engineers, and it suits projects that are lead by people with a background in computer programming. Waterfall systems tend to be the preference of small digital agencies and the internal IT departments of organisations whose primary products and services have nothing to do with software.

There is no right or wrong way to make software. Both classic waterfall and modern agile methodologies have their strengths and weaknesses. The best software engineers select a mixture of techniques from both approaches. And the best approach varies from one situation to another; influencing factors include things like team experience, the scope and scale of the project, dependencies on other people and groups, the technologies being used, dependencies on legacy IT systems, and the culture of the organisation.

I like to think of waterfall versus agile as a continuous spectrum. Individual software engineers have their own default preference as to how to manage software projects, sitting somewhere on that spectrum between extreme waterfall and extreme agile. We all interpret "agile" differently, and that's perfectly fine.

Personally, I sit somewhere near the agile end of the spectrum. But I'm no purist. For example, I think that all software projects, big and small, benefit from good documentation, something that agile methodologies tend to discourage because the writing of documentation makes no direct contribution to the writing of production-ready source code.

The recent trend might have been toward agile, but classic waterfall methodologies are not going away.