Agile to the Extreme

Feature: A review of Agile Development and XP-Extreme Programming
Motivation: Better IT project management is at core of interest in Agile Methods and XP

Many developers equate Extreme Programming with Agile Development and use both as alternatives to the classic SDLC-System Development Life Cycle. But as we shall see below classic Waterfall and SDLC methods have delivered ambivalent results. On the one hand some very spectacular successes but at the same time some very high IT project failure rates. And so on a first level, they are right - both Agile Development and Extreme Programming do represent very different styles and are even more tailored to the fast and complex pace of contemporary development. But to identify either Agile or XP-Extreme Programming as the one best way to do contemporary development is to make the same mistake that has allowed SDLC and other engineering Waterfall methods of project management be applied to any and all IT projects with decidedly mixed results. This is a repeat of the myopia of "one best way" which hampered much of organizational leadership and management theory for much of the twentieth century.

It took 60 plus years of treasties on leadership style and books on management methods by thought leaders as diverse as Henri Fayol, Alfred Sloan, James Thompson, Jay Galbraith, March and Simon, Lawrence and Lorsch and others to come to the conclusion that there is no one best way to lead or manage an organization; but rather depending on key environmental variables(uncertainty, risk, and competitive players are leading ones)and intrinsic organizational factors (capital intensity of the industry, age of the organization, maturity of the major markets are key dimensions here)there is a continuum of processes and structures by which to manage an organization. This is the Contingency Theory of Management - adjust the organizational structure and/or processes to meet the changing internal and external factors confronting the organization.

Here is one of the influential contingency theorist, James D. Thompson and his formulation of the key dimension in organizations and what that means - "Uncertainty appears as the fundamental problem for complex organizations, and coping with uncertainty, as the essence of the administrative process ...organization's form must respond to this problem ". Other theorists, notably Scott and Galbraith added other dimensions. But the key idea here is a)there is no one best structure or way to manage an organization and b)one changes organizational structure and processes to suit deep change in the organization and/or its environment.

Remarkably as the contingency theory of management built up adherents and adopters through the 60's to the 90's, right next door, the management of IT projects stayed rooted in a one best way approach - some variation on the SDLC-System Development Life Cycle. The attitude has been to add a phase or two(see Scott Ambler's Enterprise Unified Process from 2005) or broaden the view to include the portfolio of projects (see the Zachman Framework) or a general organization wide viewpoint- and these broader perspectives would allow IT projects to be managed better.

And to a certain extent, these new methods do work well for some projects because they do meet the needs and requirements of a certain class of projects. However as a one best way to IT projects - they certainly do not fit the bill. And their proponents recognize this and do not propose that their methods be applied in all IT projects. However, like those maddening broader proofs in math texts left as an execise for the reader, most IT project theorists leave it as an exercise to a)determine when and under what circumstances their method should be not be used and b)what alternative methods or simple process adjustments users should make to switch to a more effective method/process for controlling their IT Project.

Note here the important point - the IT project either has changed or has simply revealed a problem or dimension not yet taken into account. The problem for IT practitioners - how do I respond to these changed conditions. And theorists for Agile Methods and Extreme Programming both say we know how to handle this very problem of constantly changing conditions. But neither XP nor Agile methods take into account broad project metrics - does this additional feature or changed process endanger the project's ROI or time-to-market constraints. In effect, is XP's relentless refactoring and feedback/communicating with customers and Agile's constant iterative refinement are both self correcting mechanisms that will allow a project to eventually get on back on course. The question never raised by either method is at what cost in time, resources and overall benefit to the organization.

For example, Craig Harmanin his book Agile and Iterative Development argues "This deep appreciation - that building software is [more like] complex, new product development with high change rates and not predictable manufacturing - is at the heart of agile and iterative methods. Certainly another driving force is the desire to compete and win. Iterative and agile methods foster the flexibility and maneuverability to win." Strongly implied is that the solution to successful IT development is carving up the IT project into appropriate byte-sized iterations that can be successively and successfully delivered such that all clients and stakeholders are satisfied.

And so different Agile methods are in the business of providing at each iteration some mechanism for accommodating change - tasks with new priorities but even more important new tasks not a part of the original specs and requirements. As anyone who has worked in IT projects with scope creep this latter "agility" virtue can also be a major problem. The following is an example.

Take a simple control system for gas plants. Originally as designed only the master control console would be allowed to make changes to the settings in the system. Very aggressive schedules for delivery were set based on the idea that this was a star design and little or no transaction processing had to be accommodated in the nodes. Then it was decided that in a few sour gas plants it was vital for local operators be able to make changes to settings. The change seemed innocent enough to local and even top managers. And Agile methods favored this apparent "small change". But the result was disastrous because a simple central transaction processing system became a much more complex design of peer-to-peer multi-user transaction processing. The result was to turn a project on schedule and on budget into a way over budget, very late project with a lot of careers and personalities damaged in the process. Clearly Agile and Iterative are not cure-alls and in some project situations are not appropriate. .

But cure-alls for what ? Underlying all of this discussion on IT project methods is the notion that either the process is complex and difficult or that IT project delivery has not been scintillatingly successful in the past or both. Lets examine these issues in more detail.

The Nature of IT Projects

IT-Information Technology is all about managing and controlling the flow of information and control within an organization. But Information itself is complex. Information is multi-dimensional and dynamic which in turn makes IT difficult to manage well over time. Here are some of the dimensions of information:
1)It is model based. This means information is not one isolated fact but rather several datum points that are the attributes/properties of a model. That model is well described when the data can be used in a defined process to describe/explain/project the current and possible future behavior of an object or system;
2)The data are ranked. Some properties and attribute are more critical in explaining/describing the behavior of the system. These are the critical factors and/or associated dimensions;
3)The data are time sensitive because dynamic systems and their properties change over time. Having data over the right time period or window of behavior is critical to help explain/describe/project the behavior of a system;
4)The data are complete. This means enough of the critical factors or dimensions of the system are being tracked and for the right time interval and sampling frequency to most likely explain and/or control the behavior of the system within an agreed upon probability or risk/reward;
5)The data are provably accurate. This means there exists other information/data that can provide an unbroken/unambiguous links and corroboration as to how specific designated properties and attributes of the system were obtained. This means the process and all the steps in how the data were obtained , stored and subsequently processed is also complete, consistent and well described and already have been or can be replicated by others using the same procedures/processes;
6)Data have different efficiencies. Because there are many paths for obtaining, processing and storing data with different costs associated with each path and different effective explanatory/predictive capability for each data path - there are different cost/benefits or efficiencies linked to each data path;
7)Because of human variability in the ability to absorb and process information, there are different values and benefits as to how data are organized and formatted for presentation.
8)The data are political. Because value of information can become extremely high in many business and organizational settings, access to it and the ability to mange it becomes political - the question of who gets access to what and how and why.

So information is at least eight dimensional - and one of those, the political dimension, is obviously a hot potato. I have trouble navigating a car in two dimensions given there are other drivers of varying competence engaged in the same game. Imagine the complexities of bringing the right info at the right time to the right people and without breaking the bank and/or upsetting long and carefully balanced political stability. The latter political factor means emphatically that information processing is as much a cultural as technical challenge. Given that IT is all about changing data and information flows within and between organizations, is it any wonder that management of change not technical feasibility is the number one problem in IT and System development ? See our article, The MisManagement of Change, for a further elaboration of this perspective.

But many, even from the IT community, argue from the scientific and engineering perspective that IT is subject to the same discipline and standardizing influences that all the other sciences are. And so that over time, practitioners develop technologies and perfect practices that allow even IT to be controlled with a high degree of competence and confidence. Lets consider some counter arguments to this engineering approach to IT.

Many observers have advanced theories as to why for over 50 years software development has seemed to be mired in mixed competence (see Failure Rates of IT Projects immediately below). Here are four factors that recur and deter success in systems development:

1)Every software system produces critical internal process changes. Many of those changes involve modifications in how group work is done, information flows, and how decisions are made in an organization. These are changes to the basic group and organizational infrastructure involving many economic, social and political interests. Only within the last 10 years has the Management of Change process been fully recognized and begun to be controlled and applied in a systematic fashion as part of IT development process;
2)Software and Hardware technologies mature and improve rapidly. In 50 years there have been at least a dozen major process or technology led revolutions. These are revolutions because the costs, functionality and price/performance can be so compelling as to make existing systems and/or processes redundant if not instantly obsolete - if you don't change others will and gain a competitive advantage. Revolutions such as timesharing, PCs with GUI/event driven interfaces, object oriented development; Moore's Law being sustained for over 40 years have obsolete swaths of skills and displaced if not eliminated millions of jobs. Revolutions are ultimately beneficial- but they exact a toll - and among others is higher risk of failure with the new technology in its introductory phase - the so called bleeding edge effect;
3)All computing hardware (CPU, memory, disk space, communication capacity)continues to follow Moore's Law - doubling in capacity in less than two years for the same price. This is an outgrowth of Smith's Law - tools tend to be used to perfect themselves at an ever faster rate. The limits are reached when barriers in the form of some immutable laws of Nature are reached. It appears for the next ten years ahead there will be no insurmountable barriers although for the first time capital cost is rising faster than the returns. So super-imposed on the revolutions and their associated (and disruptive) killer applications, there has been an astonishingly relentless decline in the cost of computing hardware. Every 3 to 5 years the economics of computing changes for a new sector of society. The latest is communications where mobile technology will bring immediate connectivity to rich and poor nations alike in contrast with the capital intensive land-line telephone which has been reserved to the developed countries primarily. So system projects have had and continue to need to cope not just with internal organizational change but also often with external market and societal changes. This has inevitably added to the risks of projects.
4)Software development has become more complex. From centralized batch processing through separation of data and processing to heterogeneous, n-tier distributed processing - a system has evolved into a many headed hydra. It used to be simple - a single program interacting with one master control system (the OS)to the contemporary situation of many codes or applets smeared and interacting over multiple clients and servers. When a problem/bug occurs where do you start looking for it.And the rate of change of programming languages (Assembler/Cobol/Fortran to RPG/PL1/Pascal to C/Focus/Basic to C++/Small talk/Visual Basic to Java/C#/PHP/Perl to Curl/Ruby/Water), and methodologies (input/processing/output analysis, structured methods, data and information modeling, event-based design, object-oriented analysis, asynchronous and synchronous design, adaptive and agile processes) reflect the need for developers to ever widen their base of knowledge. Developers typically have to have 7-8 technical skills: HTML, XML, SQL, UML or ERD, Cold Fusion or PHP or Perl, JavaScript or VBScript, CORBA/RPC or J2EE or .NET, C/C++ or Delphi or Java or VB6/VBA, ABAP or People Script or ..... A. Russell Jones over at Devx got it right - developers need a broad base of skills. Getting the right combination of skills lined up together on even a medium scale project can be a nightmare again increasing project risk even in these days of time-boxed, protoype based development.

In sum, programming and system development is a inherently a risky business because of the nature of information itself and the ongoing rapid change that has been constantly percolating through the industry. And nothing underlines the fragility of current development processes than the record of failure in IT projects.

For a long time IT project failures were simply acknowledged but not counted. then in the mid 1980's and early 1990s Caper Jones Software Productivity reports listed failure rates in key segments like military, government, software vendors, telecommunications. But it wasn't until the Standish Group in 1994 released it Chaos Report with a clear classification of IT projects among three categories:
- successful: the project is completed on time, on budget and with all the features and functions as originally specified;
- challenged: the project is completed and operational; but over budget and over the time estimate and with fewer features and functions then originally specified;
- failure: the project is cancelled before completion.
Given the breadth of the survey and its corroborating evidence from other sources such as SPI and Dr. Howard Rubin at NYU, the staggeringly high rate of challenged and failed projects caught many practitioners off guard. Even more disturbing were some of the attributes of challenged and failed projects - time overruns averaged 222% over estimates while cost overruns averaged 189% greater than budget. Stark results. The chart below shows the slow progress being made in IT project management effectiveness.

Again this appears to point to a recurrent theme in IT project management - that many of the issues associated with project failure are people and managerial issues - not technical issues. And as we shall see below there seems to be evidence that handling the people and management of change issues are consistently one of the greater risks for IT projects.

The net result of these hgh project failure rates was the call by consultants for better project control. In his book Strategic Project Office, J.Kent Crawford on page 7 notes that Gartner in 2001 proposed in one of its strategic assumptions that "IS Organizations that establish enterprise standards of project management, including a project office with suitable governance, will experience half as many project cost overruns, delays and cancellations as companies that do not do so. They also note that the software development project as presently managed is 170-180% over budget." In effect, project management goes to the top of the list and the executive suite as a means of managing IT projects risks.

JohnMcManus (see references below)discusses the causes of the project failures previously noted. Interestingly he see the following breakout for failures in IT Projects:
Managerial issues account for 65% of failures
Inappropriate project structure and channels of communication
Inappropriate resources - poor skills and knowledge mix
Inappropriate estimates of time and resources
Inappropriate planning methods - poor scheduling and tracking
Inappropriate user buy-in
Inappropriate risk management

Technical issues account for 35% of failures
Inappropriate software requirements
Inappropriate technical design
Inappropriate development and testing tools
Inappropriate technical documentation
Inappropriate technical reviews and quality audits
Inappropriate technical support

More interesting is the changing prognostication from the Standish Group based on its Chaos Report analysis. First, here are the ten ingredients for an IT project highly likely to succeed.
1)Reduce the requirements to the stark minimum like in Extreme Programming-XP;
2)Provide constant communication on project progress/feedback like in XP;
3)Use standard code infrastruture - either packaged software or well tested standard lib aries;
4)Have a good project manager;
5)Have a strong executive sponsor (the latter 2 points Standish elaborates in detail);
6)Use an iterative process for deciding on next phase features ;
7)Use good project tools (again Standish says a lot here);
8)Use no more than 6 people on the project:
9)Allow for completion in 6 months or less;
10)Do not spend over $750,000.
Now the last 3 of these requirement may seem very narrow; but they are based on Standish's stats - small, low budget, short duration projects have significantly higher success rates. So how do users fit their projects into these "best chances of success" constraints ? By also considering the causes of project failure.

Standish again has isolated 3 factors: lack of end-user involvement, failure of top executive support and poor project estimates of risk and resources. To an extent, these are inverses of the some of the 10 project success factors and thus give a reading on which of the 10 are most important.

However, of equal importance is that Standish substantiates John Mc Manus thesis - that managerial issues and not technical complexity predominate as causes of IT project failure. In the next phase of our analysis we shall put these ideas under a little tighter scrutiny.

This first phase of the analysis of Agile Methods and Extreme Programming has raised a fundamental issue. The Nature of Information and evolving complexity of IT projects means also there is not one best way to manage IT projects. It also states that both Agile Methods and Extreme Programming can be effective in certain classes or kinds of IT projects (and the next part of this review will attempt to identify dimensions and criteria used to choose either Agile Methods and/or Extreme Programming. Finally, this latter section will also look and see how historically the two methods moved thinking in IT project management for one standardized Waterfall model to a discussion of "heavy" and "light" project management - and therefore moved the discussion firmly away from a view that the SDLC, if appropriately expanded in scope, could handle any IT project.

Contingency Theory of Management
Jay Galbraith - 1973 Designing Complex Organizations, Addison-Wesley.
Jay Galbraith - 1977 Organization Design. Addison-Wesley.
Maura F.Guillen - 1994 Models of management: work, authority, and organization in comparative perspective. University of Chicago Press.
Paul Lawrence and Jay W. Lorsch - 1967 Organization and Environment. Harvard Univ. Press.
Richard W.Scott - 1987 Organizations: Rational, Natural and Open Systems. Prentice-Hall
Yehouda Shenhav - 1995 'From "chaos" to systems: the engineering foundations of organization theory 1877-1932'. Administrative Science Quarterly 40: 557-585.
James D.Thompson - 1967 Organizations in Action. McGraw Hill.

IT Development
Craig Larman - 2002 Agile and Iterative Development: A Manager's Guide, Addison-Wesley
John McManus - Risk Management in Software Development Projects
Jason Harvat - Project Management Methodologies, J.Wiley & Sons
Erik Keller - 2004 Technology Paradise Lost - Manning Publications
Chromatic - 2003 Extreme Programming Pocket Guide - O'Reilly
Standish Chaos Reports:IT projects - http://www.standishgroup.com/sample_research/index.php