I got some smart comments from Henry over at The Modern Middle Manager, and went over to take a look. He had a great post on an almost-failed eCRM implementation, and it started me thinking.
About 70% of the billable work Ive done in the last five years has been work on failing technology projects. These are projects from simple websites to the project Im on a now, a multi-million dollar customer data warehouse for a major corporation. (The project is Really Far from my house, and while Im glad for the work, Im getting tired of commuting and anyone in Greater LA who has a troubled tech project ought to email me )
Almost all of these projects got into trouble for similar reasons. Lack of planning, lack of clear alignment among the stakeholders, a failure to follow rudimentary systems development processes, vendor deceit, and most of all, hubris.
Hubris is the most important thing, because it is what allows the project sponsors and leaders to believe that what theyre doing makes sense, even in the face of contradictory best practices and evidence.
Hubris in the technology world typically looks like this:
A visionary engineer builds a tool or product. He partners with or hires, or is hired by an aggressive marketing visionary who takes the actual capabilities of the tool or product and ties them to a business need in the most visionary and optimistic way. They hire an energetic PR person, who gets them press in Wired (or the late lamented InfoWeek), where it gets read by an executive at an organization that has the business need targeted. The executive groks the vision, and calls the vendor, who sends a sales executive and sales engineer to meet with the organization, and sell a vision of a seamlessly integrated, real-time view of the company processes, or customers, or whatever.
Now the bigger the vision, the bigger the ROI, and the interests of all parties are well-served by revolution, rather than evolution. So the decision is made to junk the existing (badly architected, lashed-together, obsolete, barely-functional) systems, and build something new from scratch.
Heres where the problems really begin.
Management, planning, analysis, and documentation rise in importance and as a necessary percentage of the project budget as the scale of the project increases.
Examples: I just designed and a friend built a small web-referral monitoring and reporting system; its a database of about seven tables and four web pages. Meeting with the customer took about two hours, I wrote about ten pages of documents (including two pages of database ERDs, one page of flowcharts, and seven pages of functional description) in about a day, and the developer worked for a week to build it. So roughly 20% of the effort was in overhead.
Ive never worked in aerospace, but Im told that on major aerospace software projects that the overhead is approximately 60% of the total budget; this is required because a) the projects are extremely large; b) releasing buggy software and iteratively working it out the Microsoft solution — is not an option; c) the management and reporting processes are extremely formal and structured, because thats what has worked on large scale engineering projects such as Apollo in the past. That system is brutally inefficient, and probably not a lot of fun for the people involved, but it works.
The problems begin to arise when project sponsors try to take projects of near-aerospace levels of complexity and apply the kind of management techniques I used in designing a four-page, seven-table web tool.
Most non-aerospace organizations lack the expertise and discipline to manage huge projects. Many of them claim to, however.
The best defense, Ive learned, is to manage the scale of the projects, rather than trying to change the management culture of the organization. This doesnt mean that you have to abandon far-reaching systems as a goal. It just means you have to break them up into littler systems that can be developed within a reasonable span of control. Personally, I tend to think that a project budget of $750,000 and a project term of four months is about right. You can do useful work and still have a small, agile team doing the work.
Henry came to the same conclusion:
Don’t create one massive project, do mini-projects and stick to a rollout schedule. We changed too much too fast and tried for too many payoffs at once (in sales, client management, operations). That’s the number one reason the end-users got irritable and the reputation of the system was trashed almost immediately.
My personal project bible says much the same thing.
If youre interested in this stuff at all, go read Rapid Development: Taming Wild Software Schedules, by Steve McConnell. Its a terrific book; Ive given about a dozen copies away to clients and team members.
OK, for the non-geeks in the crowd, whats the relevance of this?
Traditional command-and-control government programs are like large software development projects; they are insanely complex, and to succeed, they require incredibly tight control over the participants. This both requires a huge overhead of people doing the controlling, and a culture of tight control as well.
And like large software projects, they fail badly. The hysterisis the time lag between action and output is so large that its almost impossible to change or adapt. Small projects fail well; they have low hysterisis, and so when problems arise, they can adapt in response to real conditions.
As weve talked about race, one of my issues is not that change needs to happen; its that the breadth of change is so great that making it requires the kind of cumbersome, likely to fail or have unintended consequences programs that Im discussing here. Im trying to see a way through the problem that uses smaller more nimble, more effective programs.
Not only for dealing with issues of race, but for dealing with the array of issues that we confront as a society, and that, as a liberal, Id like to use the power of society and the government to improve.