Chaordic Development: A New Direction for Large-Scale IT Initiatives
Seeking order in a sea of chaos? Chaos, in the case of an information technology (IT) project, means being caught up in the miscommunications, politicking, competing agendas and confusion inherent in many large-scale initiatives. Better tools, methods, training and technology have not improved our ability to deliver high quality, on-time projects. New and complex technology has even contributed to poor performance by creating a moving target for design and implementation teams.
The issues undermining these projects will escalate unless executives address the underlying institutional weaknesses causing these failures. This requires all relevant and affected parties to agree on a set of shared principles that inspire disparate business units to work towards a common project goal. We must acknowledge and leverage the inherent chaos within these environments. In other words, we should apply "chaordic" organizing principles to large-scale IT initiatives.
Managing at the edge of order and chaos is emerging as a mega trend in the IT industry not by intent or by design - it is just the natural state of people and projects. Chaos will not go away, and neither will the need for some semblance of order. But learning how to manage and leverage these seemingly disparate states concurrently will lead to successful projects, and success is one mega trend that must begin to emerge.
Where Did We Go Wrong?
Since the early days of computing, business and technology analysts have struggled to deliver new and better information systems. As the decades passed, small problems grew into insurmountable ones. Consider this scenario. A group of business unit managers wants a system replaced, but cannot articulate why or how. These managers just know that they are unhappy with the current situation. Other business units have different ideas. Some want a package, while others insist that a shift to e-business is inevitable. In response, IT executives have hired a group of management consultants to define requirements for a replacement system. But the consulting firm is not coordinating their efforts with the corporate architecture group or related business units.
To complicate the situation, the existing system support team, depleted by attrition, thinks any redesign effort lacking their input will fail. They alone understand how the current business environment has been automated. Yet repeated replacement efforts have ignored their potential contributions. Complicating this impossible situation is the fact that the entire initiative lacks sponsorship. Executive sponsors just want everyone to work things out and keep them out of it.
If this sounds familiar, it’s no coincidence. Problems like this are commonplace. This assertion is validated by an oft-cited 1995 Standish Group International study that found more than 80% of IT projects being delivered late or not at all, and with 31% percent canceled outright. An ongoing Cap Gemini study shows that this situation has not improved over the past four years. As of fall 1999, 43% of all Year 2000 project completion dates had slipped into the fourth quarter. Just three months prior, only 16% of those projects were showing fourth quarter completion dates.
Such findings clearly indicate that project teams lack the cohesion necessary to bring major information technology initiatives to a successful and timely conclusion. Reviewing the top five challenges cited by Standish Group study participants helps clarify why project deadlines and quality have suffered. These reasons included 1) a lack of user involvement, 2) inadequate executive support, 3) no clear requirements statement, 4) improper planning and 5) unrealistic expectations.
A lack of user involvement indicates that parties needing to reach out to each other are not doing so. Inadequate executive support implies a lack of top management sponsorship and leadership, required elements in a major IT project. Unclear requirements stem from poor communication among project stakeholders. Improper planning suggests that participants did not spend enough time articulating critical success factors and related milestones for the project, while unrealistic expectations signal poor communication among project participants.
Pouring more technology, more consultants and more money into a project will not make these problems disappear. Some people may want to outsource the project, but this is not a panacea either. In the absence of defined organizing principles, outsourcing is likely to exasperate an already bad situation by shifting the responsibility for critical tasks away from key project stakeholders. We must find a way to allow good people to deliver quality IT solutions.
What can Management Do?
Piecemeal solutions will not address the complexities highlighted above. These issues are systemic and cut to the heart of what is wrong with many institutions today. Executives cannot mandate disparate groups to cooperate. Neither can they mandate success. Sponsors must create a governance structure whereby relevant and affected parties are motivated to work towards a common goal and a successful project.
Creating this governance structure means drafting a formal "constitution" that gives stakeholders a framework in which they and their project can thrive. In other words, management must establish a chaordic organization to govern the success of major IT projects. According to Dee Hock, founder and CEO emeritus of Visa International and coiner of the term, a "chaordic" organization is:
- any self-organizing, self-regulating, adaptive, non-linear, complex organization, the behavior of which harmoniously exhibits characteristics of both order and chaos;
- an entity whose behavior exhibits diverse patterns and probabilities not governed or explained by the behavior of its parts;
- an entity in harmony with the fundamental organizing principle of nature and evolution [1].
Chaordic principles are increasingly being applied to a variety of institutional organizing efforts. A sampling of these projects are listed in Table 11. The chaordic organizing process ensures all relevant and affected parities to function as an integrated unit, working towards a common goal under a formal governance structure.
Table 1: Sample chaordic organizing projects
| 1. United Religions Initiative |
| 2. Appleseed Foundation |
| 3. Community Alliances of Interdependent Agriculture |
| 4. Northwest Atlantic Marine Alliance |
| 5. Society for Learning |
| 6. VVALEO Healthcare Initiative |
| 7. Geographic Data Organizing Initiative |
Chaordic organizing principles can be applied to IT projects to nullify many of the problems highlighted in the Standish Group study. The steps needed to accomplish this are shown in Table 2.
It is possible for relevant and affected parties to come together and outline individual interests, yet not come to a shared purpose for the project. If this happens, project sponsors will have been spared from wasting critical energy and capital on a failed initiative due to the lack of a common vision among key participants. At that point, relevant and affected parities would be free to reorganize into teams that could tackle alternative solution scenarios. Either way, senior managers would have discovered early in the project that they had a problem, well before they ended up spending $20-30 million on an initiative that never came to fruition.
Table 2: Creating a chaordic alliance for a major IT initiative
| 1. Gather all relevant and affected parties contributing to or benefiting from the project. |
| 2. Determine each participant’s stake in the outcome of the project. |
| 3. Agree on a common purpose and set of principles to govern the group’s behavior on the project. |
| 4. Identify the people needed to make the project a success in accordance with the project’s purpose and principles. |
| 5. Create an overall project strategy to meet the purpose and principles for the project. |
| 6. Establish a written constitution to govern project participants for the life of the project and the life of the system produced by the project. |
Applying Chaordic Principles to IT Projects
Project teams need to proactively address the litany of problems outlined that plague IT projects. To accomplish this, executives should sponsor a series of working sessions with stakeholders to create a common set of project principles and governance structure. To more fully describe how the creation of a chaordic IT project team might unfold, we will revisit the problem project scenario that I outlined earlier.
1. Gather relevant and affected parties to initiate the project.
Turning ill-defined project teams with conflicting goals into cohesive and effective work units requires stakeholders to participate in a project kickoff session. Stakeholders include sponsors, users, designers, developers and support teams for the proposed system. This would include representatives who may be supporting systems that the project may replace. In our earlier scenario, stakeholders were not working together and, in some cases, not even speaking to each other.
Executive sponsors, business unit representative, current systems support personnel, design teams, information architects and consultants should convene a meeting to self-organize. If certain relevant and affected parties refuse to participate, the project may fail. Getting participants in the same room at the same time is a major hurdle and an initial organizing meeting could take some time to materialize. The agenda for this meeting is discussed in step two.
2. Determine each participant’s stake in the effort.
Relevant and affected parties should share their vision with respect to the pending initiative. Business unit sponsors should summarize why the team was assembled and who is in attendance. Executive participants must be careful not to dominate the meeting or the dialog. All participants must be viewed as equal contributors. Open communication and honesty is essential.
One ice breaking technique involves having each participant pair up with another stakeholder to share their ideals for the project. Each individual participant must then share his or her partner's vision with the group. This will begin to open up the creative thinking process. Individual visions can be documented, grouped into common categories and serve as input to the development of project purpose and principles. This session allows the team to assemble a more complete set of ideals than any given participant could individually bring to the table.
Provided the group agrees to continue working together towards a common set of visions, it should choose representatives to form a project alliance design team. This team will not design the system, but will create an infrastructure that will govern the process under which the systems design and implementation process can flourish. The project alliance design team, which would include representatives from each stakeholder constituency, will be responsible for drafting the project purpose, guiding principles and governance structure.
3. Draft a common purpose and set of principles.
Defining a purpose and set of principals for the project alliance is a challenge for most project teams. Agreeing on a purpose to guide stakeholder participation is the first step. Each stakeholder will have unique and potentially conflicting goals. In the scenario that we are examining, the following statement of purpose could serve as an outcome of this step in the process:
Build, deploy and support an integrated system that satisfies customer and greater community requirements, as it relates to corporate marketing, sales and support functions, through the ongoing use of leading edge technology.
The term "greater community" suggests broad sensitivity to environmental and community concerns that go beyond your immediate customers. The team must now define a set of principles to govern this initiative. Sample principles for this project scenario are outlined in Table 3.
Table 3: Sample project alliance principles
| 1. Put the customer first in all project decisions. |
| 2. Accommodate the environmental and societal concerns of impacted communities. |
| 3. Meet the collective needs of the marketing, sales and customer support business units. |
| 4. Integrate functional areas from a business, data and technical perspective. |
| 5. Utilize, as appropriate, off-the-shelf, new and existing data, business rules and technology. |
| 6. Incorporate legacy data and business rules into the design and development function. |
| 7. Allow any party to join the project if they comply with project purpose and principles. |
| 8. Stress integration with related business areas that will not directly utilize the target system. |
| 9. Position the design and development feedback cycle to allow input from all stakeholders. |
| 10. Create a phased design and implementation plan that delivers interim project deliverables to benefit stakeholders. |
| 11. Maintain open sharing of project status for any party requesting that information, as long as it does not compromise privacy or security rules. |
| 12. Create a system that can be maintained effectively and efficiently for a minimum of ten years. |
| 13. Decision-making is necessarily vested in and functional activity performed by the component of the project alliance team that is most directly affected. |
| 14. Quarterly meetings will be held to see if the project is adhering to its purpose and principles. |
| 15. The project can be canceled, any time, by an 80% vote of all project alliance members. |
Because each principle can call into question the purpose or other principles, the team must periodically revisit the purpose and principles for the project. If the team cannot agree to a set of principles, they may halt the project at that point.
4. Identify the people and the project structure needed to make the project a success.
This step identifies internal and external teams needed to complete the project and assigns personnel to those teams. In our problem project scenario, the following teams would apply.
 | Project integration and oversight team. This team manages the project, resolves conflicts, seeks compromises and verifies that the team adheres to purpose and principles. |
| Funding Team. This group is responsible for obtaining project funding from executives and managing against a fixed budget. |
| Architecture oversight team. This team ensures that the target business, data and technical design process maps to institutional and / or preferred standards. |
| Business unit requirement definition team. This team defines business requirements based on stakeholder input each business unit and should focus on integrating functions across those business units. |
| Data specification and integration team. This team assesses current and target data requirements and then designs a common data architecture to be used by the target system. This step would accommodate legacy data migration issues. |
| Target specification design team. This team turns the business requirements into a target system specification document that reflects the ideal target system design. |
| Legacy system, package system and target design mapping team. This team outlines where target system design specifications should be derived from and requires a high degree of specification mapping experience. |
| Technical specification team. This team defines the technical environment and should include Internet, e-business, legacy technology and transition management skills. |
Participants for each team are drawn from the population of stakeholders on the project. This includes business and technology analysts, legacy and new design experts, outside consultants and other specialists. New teams can be created and existing teams can be disbanded as long as they adhere to project purpose and principles.
5. Establish an overall project strategy.
Once teams have been assigned, these teams should draft an overall strategy for the project. This is done prior to finalizing the project constitution, because the strategy is a part of the constitution. The project strategy identifies each design team, defines the design specification protocol, outlines project work tasks and establishes desired project timelines. The strategy also should list participating business units, impacted legacy systems and target software packages to be considered during the design process.
6. Create a constitution to govern project participants.
At this point, the project alliance may make the difficult decision of whether to establish a separate legal entity to own the initiative. While many corporate entities may consider this unnecessary, it has the advantage of removing internal politics from the project. Either way, a constitution should be drafted to incorporate all of the work completed in steps one through five. This formal document has a place for stakeholders to sign their name, committing to the purpose, principles, roles, design concepts and strategy for the project.
The constitution can be changed or terminated by a vote of 80% of the participants. Anyone not adhering to this constitution would be eliminated from the project. If a business unit decides to drop out of the project, the constitution may need to be updated. Once named to this document, participant commitment levels to the project are binding. This approach also embodies a strong sense of trust among all participants.
Chaordic Shift: Essential to IT Success
IT projects expose systemic failures in harsh and unforgiving ways. Consider the disastrous efforts by the IRS to modernize their aging tax systems. According to an October 1999 New York Times article2, antiquated systems and a never-ending flood of impossibly complex tax laws have left billions in uncollected tax revenues. Congressional lawmakers and the people responsible for turning tax laws into operational systems need to lock themselves in a room and draft a chaordic alliance to address this issue. This is unlikely to occur in the politically charged atmosphere of Washington DC.
On the other hand, opportunities for chaordic alliances within and across many industries abound. Area code proliferation is a disaster in the making within the telecommunications industry. IT teams will ultimately be tasked with the expensive and time consuming task of expanding area codes, but this should be a last resort. The case against area code expansion must be made to industry leaders via a chaordic alliance between the telecommunications oligarchy and the IT teams servicing them. If expansion is imminent, then the alliance could draft a constitution to meet this goal. Chaordic concepts could also be used to integrate the plethora of old billing systems currently dragging down domestic telecommunication companies.
Industry leaders can continue down a path of making unilateral decisions and throwing them over the wall to technology teams –a failed strategy of the past. Or they can create chaordic alliances needed to deliver the cohesive, holistic solutions so essential to our increasingly complex world.
- Hock, Dee. "Birth of the Chaordic Age".
(Berrett-Koehler, 1999) - Johnston, David Cay. "I.R.S. Is Allowing More Delinquents to Avoid Tax Bills".
(New York Times, October 10, 1999)