Martin Colebourne

Software development projects are often complex and chaotic, with huge communication challenges. Despite huge investment and focus, delays and failures are still fairly common occurrences.

In contrast, many traditional industries seem to have formulated very well understood and reliable processes. However, rather than copying these approaches, the software industry is increasingly turning to a brand-new way of doing things: Agile.

Why are traditional methods not appropriate for software and why does Agile make sense?

This article explores the question by looking at how software development compares to three other industries: vehicle production, architecture and filmmaking.

VEHICLE PRODUCTION

Vehicle production can be seen as having three stages:

• Detailed business planning: establishing what kind of vehicle to produce, the size, key requirements, the markets it will be sold in and the likely market size.
• Design, engineering and prototyping: an extensive period of design, engineering, computer simulation and physical prototyping. This phase includes extensive testing – assessing performance, safety, reliability and resilience to extreme conditions. The design phase also includes the sourcing of components from suppliers, design of the production lines themselves and planning every aspect of the assembly process.
• Production, distribution and sales of the vehicle. Once this phase begins, further changes to the design are very limited. The majority of the design – especially the core engineering – remains constant.

ARCHITECTURE

Architectural design covers a huge range of project types, so let us consider just one example: the design and construction of a single large office block.

• The first phase is one of business planning, which includes the selection of an architectural partner, site acquisition, and managing legal issues such as planning permission.
• Once the basics are in place, then the architectural design work can begin. This works up from initial concepts into progressively more detail. Hand-drawn sketches and small-scale models are common early on, with CAD used to draft detailed designs later on. Prior to construction, detailed blueprints will be drawn up and the logistics of the construction process carefully planned to ensure that the build can proceed smoothly.
• The construction process follows the plans, adjusting where necessary to cope with issues with manpower availability, bad weather and other problems as they arise. The detailed design of the services and internal features may proceed alongside the basic construction and will be more subject to change.

Here is a visualisation of that process. The overall phases are familiar from the vehicle example, but the curves are somewhat different. Design completeness doesn’t rise as steeply or reach as high prior to construction. This reflects the fact that details of services, internal spaces and plans for the fit out of the building can change considerably even during construction.

Just as in the vehicle example, production completeness only rises once construction begins.

The largest costs are incurred during the final phase, where the number of people is higher and there are costs for plant hire and raw materials. The cost of change therefore shows a spike at the start of construction – changing foundations and knocking down walls is expensive. However, a degree of change is quite feasible because this is essentially a custom build of a single example – very different to a production line. In some ways, this is a better model for software development, since code need only be written once – there isn’t a mass production phase.

FILM MAKING

Filmmaking can be divided into five phases:

• During the Development stage the basic story is selected and worked up from an initial treatment into a screenplay. These elements are used to pitch the idea to interested parties in an attempt to gain funding to make the film.
• During Pre-production the film is designed and planned in detail. The screenplay will be revised, storyboards are used to illustrate the scenes and test the flow of the film. Animatics may be produced to visualise the storyboards and script in a low-fi video form. The cast and crew are hired and production is planned in detail. Locations are found, scenery is planned and built, costumes are designed and made.
• The Production phase is where the actual filming takes place. This is typically compressed into the shortest possible time period, juggling the availability of the cast and working around issues with weather and lighting.
• The Post production phase involves a much smaller set of people. This is where the film is edited, sounds effects are created, visual effects added and the score written and performed. There may be changes to the film at this point, with scenes deleted, or edited in a new way, to save time, or adjust the flow of the film.
• Once the film is complete, then Distribution begins, with copies made and sent out for screening.

At a high level, the process for Filmmaking may seem similar to the previous two examples, but the chart shows how different it really is. It would be tempting to assume that the Production phase is similar to the Production or Construction phases for the previous examples, but this is not true. Design completeness rises gradually over the course of the project, but significant change can still take place during Post-production. Production completeness only starts to rise during Post-production – the material is gathered during Production, but the film is not actually put together until later. This makes it very different from the previous two examples.

The largest cost occurs during the production phase: the number of people involved is much higher, and locations, sets, costumes and actors are all expensive.

The cost curve shows two spikes – one at the start of Production and a more important one at the end. This represents the fact that once Production plans are set, there is a significant cost in changing them. However, it remains possible to make changes during Production, even rewriting scripts and changing elements of the storyline. In contrast, once Production finishes, the cost of making major changes becomes much higher – this reflects the fact that the core material the film will be made from has been captured. It remains possible to rework and rearrange scenes using the available material, but going back and reshooting scenes is prohibitively expensive.

SYNTHESIS

Although there are many differences in their detail, there are clearly common underlying threads between these three examples. I believe that there are three key factors at work:

• Production completeness doesn’t begin until late in the process.
• One of the later phases is dramatically more expensive than the rest and this produces a step in the cost of change function.
• There is, in essence, only one chance to get things right – the finished vehicle must sell, the building must work and audiences need to like the finished film. Waiting until the artefact is nearly complete in order to test it means waiting until it is too late to fix it.

These factors drive a particular approach to the process: an emphasis on up-front design and planning in order to minimise the risk of doing the wrong thing in the expensive phase and producing something that doesn’t sell.

WHAT IS DIFFERENT ABOUT SOFTWARE?

If the process for traditional industries is driven by these key underlying factors, the question is whether software development shares those same features. This section explains that software development has a number of key differences that set it apart from traditional industries.

LACK OF A PHYSICAL DELIVERABLE

Software is fundamentally digital – it is made up of lines of code. Nothing physical is produced. This is quite unlike the vehicle design or architecture cases, where the end result is all about physical product. Even in the filmmaking example, where the end result is not a physical deliverable, the Production phase involves a lot of physical production and activity – making sets and costumes and actually acting out and shooting the scenes. In the three examples, it is the phase where the most physical activity happens, or physical production occurs that is the most expensive. Software production doesn’t really have a parallel – none of the phases require a significant quantity of manpower and raw materials, so there isn’t a large spike in cost.

INFINITELY EDITABLE

Software is very fluid: it can be altered and reworked repeatedly. And this can be done piecemeal: a piece of code in the finished product could have been written on day one and remained unchanged, written on the last day, or rewritten hundreds of times during the process. This means that the construction process happens almost continuously throughout the bulk of the project.

COMPLEX AND INTANGIBLE

Software is interactive and non-linear, often with thousands of possible internal states. This makes it highly complex. At the same time, much of what is going on is intangible – it occurs in the mass of code that exists ‘below the waterline’, with no direct representation in the user interface. It simply isn’t possible to ‘open the hood’ and see what is happening inside.

INNOVATIVE

The majority of design projects in established industries are about creating a new version of something familiar. The aim is to make a better version, a cheaper version, or one that is designed to serve a particular purpose. Elements of the design may be innovative, but the broad thrust of the enterprise is familiar.

In contrast, much of software development is innovative – it is not aimed at producing a piece of software that is a competitive version of a well-established class of products. Instead, most software uses existing elements to create a more or less novel solution. This means that the requirements for are almost always very poorly understood and exist in an environment of uncertainty.

LACK OF PACKAGING CONSTRAINTS

In other industries, there are powerful packaging constraints at play that force careful planning and testing of how the whole artefact comes together:

• In the vehicle design case engine, transmission, suspension, electrics, fuel tanks and so on need to fit within a single small space and leave space for passengers and luggage.
• A building must fit within its site, provide accessible entry and safe emergency exit for visitors, space for air conditioning, power, water and sewerage etc. All whilst fulfilling the brief for the purpose of the building.

In contrast, code can be constructed piece-by-piece and then put together without necessarily needing to modify the elements. Although there can be issues with unexpected interference between parts, or with emergent performance challenges, these are typically dealt with after the fact, rather than intensively designed up front.

LACK OF LEGAL AND PROFESSIONAL CONSTRAINTS

Things often happen because it is possible for them to happen, even if it is clearly undesirable. In many industries, a framework of legal constraints, government-backed testing, professional certification and other measures has been developed to enforce good behaviour.

In comparison with the other examples, there are very few such constraints on the creation of software. This permits a much greater degree of latitude. In software, whether it works (or appears to work), is generally sufficient.

SOFTWARE PROCESS

The previous section outlines many ways in which software might be a special case. How does this impact the process of making software?

In the past, the answer to this would probably be: not much. Software design and development methods looked much like those for vehicles or architecture, with business planning, design, coding and a distribution phase where software was copied onto disk and sold. In this approach, coding was seen as akin to production or construction.

The chart for this approach would look something like this. At first glance it seems familiar enough, but there are some very important differences.

As discussed in the previous section, software is digital and doesn’t need lots of manpower and raw materials in order to create it. As a result, there isn’t a huge spike in cost for any of the phases. Coding is the most expensive phase, but the difference is not that dramatic.

Since code is infinitely editable, the step in the cost of change occurs at the end of the coding phase (where the finished code is copied and distributed), not at the start. This is very different to the other examples, where the step change happens relatively early.

Although there is a gap between design completeness and production completeness, the two lines remain stubbornly close together throughout. Work done in the early phases tends to focus on the set of requirements, the user interface and high-level architecture of the solution. There is not an attempt to ‘design’ the details of the code. In fact, given the complexity of code and the fact that many projects are inherently innovative, its not clear that there is any meaningful sense in which you could ‘design’ the code. This means that the design completeness and production completeness curves never separate that much from each other.

At the same time, there are few constraints to enforce investment in detailed up-front design. Packaging constraints are either non-existent, or usually safely dealt with later. And there are few, if any, legal and professional standards governing how software is created. As a result, there is a tendency for the depth and quality of up-front design and specifications to decay if not actively reinforced.

SYNTHESIS

The underlying misconception with the classic approach to software development is that coding is like the production phase in other industries: that the act of programming involves executing on a plan. This is simply wrong. Any plan is just the barest sketch of a solution. Programmers spend the coding period constantly generating creative solutions to achieve the desired end result. This means that coding is a form of design.

This implies that software development is best considered to be ALL design. It is only at the end of the process that we know what it was that we needed to build. But since code is digital and there is no physical deliverable, once the design is complete, so is the project. Copying the program onto disk for distribution is trivial.

From this point of view, the time spent in the Analysis and Design phase is largely wasted – it doesn’t deliver the benefits that it does for other industries, and it delays the start of coding, which, as we’ve seen, is best considered to be a part of the overall design process.

ITERATION

There is one other important difference with most software projects that makes the diagram misleading. They don’t stop at the end – instead, the process begins again to create version 2.0. The next version typically builds on the code of the previous, adding to it, reworking parts and removing some others.

This is completely different from the other examples:

• You don’t create your next vehicle by dismantling the previous cars and reusing and recycling the pieces.
• You don’t construct your next building using parts of the last one.
• And you don’t create a Hollywood sequel by recycling the plot and footage from the first film. (On consideration, maybe there are some examples that get pretty close!)

With the move towards continuous integration and deployment, server-side technologies and software as a service, this difference is becoming even more acute. Software development is becoming a continuous, never-ending process. Updates to the software can be released to clients every month, every week, or even multiple times a day.

This is vitally important, because it means that, unlike more traditional industries, there is more than one chance to get things right.

AGILE

In summary, these differences make software development radically different from other industries:

• There isn’t a huge cost spike that must be mitigated by intensive up-front design, prototyping, simulation and testing.
• Software is sufficiently complex and intangible that it isn’t really possible to fully design up front and then just implement.
• Projects are frequently innovative, such that the right solution is not clearly understood at the beginning of the process.
• Software can be iterated freely, allowing mistakes to be discovered and fixed, and areas of opportunity to be pursued in future releases.

It is these properties that have prompted the gradually development and uptake of Agile methods. These approaches take the differences inherent in software development and use them to create an advantage.

Here is a chart illustrating an Agile process. Note that it is composed of lots of short phases (and the implication is that the number of these is not fixed – it continues to the right). Distribution of the product begins early – in other words, early versions, with limited functionality begin to be made available as soon as possible. Either for assessment by internal stakeholders, or for early testing by clients.

Design completeness runs ahead of production completeness, but the two lines never diverge significantly – a recognition of the idea that the design process (whether visual, interaction, or code) is ongoing throughout.

Note that spikes in the cost of change are largely eliminated by instituting continuous distribution of the software (for example, by running it server side, so that users always use the latest cut).

By taking this iterative approach, software can be developed bit by bit and tested to see what works and what doesn’t. This is a huge departure from traditional methods, that is permitted by the underlying characteristics of software.

CONCLUSION

Software has significantly different characteristics to other industries. This makes the copying of traditional methods unwise. Agile is an attempt to turn the unique features of software development into an advantage, by changing the form of the underlying project from a single iteration of planning, designing and building, into lots of small iterations, each one delivering a working, testable piece.