I got some smart comments from Henry over at The Modern Middle Manager, and went over to take a look. He had a great post on an almost-failed eCRM implementation, and it started me thinking.
About 70% of the billable work I’ve done in the last five years has been work on failing technology projects. These are projects from simple websites to the project I’m on a now, a multi-million dollar customer data warehouse for a major corporation. (The project is Really Far from my house, and while I’m glad for the work, I’m getting tired of commuting and anyone in Greater LA who has a troubled tech project ought to email me…)
Almost all of these projects got into trouble for similar reasons. Lack of planning, lack of clear alignment among the stakeholders, a failure to follow rudimentary systems development processes, vendor deceit, and most of all, hubris.
Hubris is the most important thing, because it is what allows the project sponsors and leaders to believe that what they’re doing makes sense, even in the face of contradictory best practices and evidence.
Hubris in the technology world typically looks like this:
A visionary engineer builds a tool or product. He partners with or hires, or is hired by an aggressive marketing visionary who takes the actual capabilities of the tool or product and ties them to a business need in the most visionary and optimistic way. They hire an energetic PR person, who gets them press in Wired (or the late lamented InfoWeek), where it gets read by an executive at an organization that has the business need targeted. The executive groks the vision, and calls the vendor, who sends a sales executive and sales engineer to meet with the organization, and sell a vision…of a seamlessly integrated, real-time view of the company processes, or customers, or whatever.
Now the bigger the vision, the bigger the ROI, and the interests of all parties are well-served by revolution, rather than evolution. So the decision is made to junk the existing (badly architected, lashed-together, obsolete, barely-functional) systems, and build something new from scratch.
Here’s where the problems really begin.
Management, planning, analysis, and documentation rise in importance and as a necessary percentage of the project budget as the scale of the project increases.
Examples: I just designed and a friend built a small web-referral monitoring and reporting system; it’s a database of about seven tables and four web pages. Meeting with the customer took about two hours, I wrote about ten pages of documents (including two pages of database ERD’s, one page of flowcharts, and seven pages of functional description) in about a day, and the developer worked for a week to build it. So roughly 20% of the effort was in ‘overhead’.
I’ve never worked in aerospace, but I’m told that on major aerospace software projects that the ‘overhead is approximately 60% of the total budget; this is required because a) the projects are extremely large; b) releasing buggy software and iteratively working it out – the Microsoft solution — is not an option; c) the management and reporting processes are extremely formal and structured, because that’s what has worked on large scale engineering projects such as Apollo in the past. That system is brutally inefficient, and probably not a lot of fun for the people involved, but it works.
The problems begin to arise when project sponsors try to take projects of near-aerospace levels of complexity and apply the kind of management techniques I used in designing a four-page, seven-table web tool.
Most non-aerospace organizations lack the expertise and discipline to manage huge projects. Many of them claim to, however.
The best defense, I’ve learned, is to manage the scale of the projects, rather than trying to change the management culture of the organization. This doesn’t mean that you have to abandon far-reaching systems as a goal. It just means you have to break them up into littler systems that can be developed within a reasonable span of control. Personally, I tend to think that a project budget of $750,000 and a project term of four months is about right. You can do useful work and still have a small, agile team doing the work.
Henry came to the same conclusion:

Don’t create one massive project, do mini-projects and stick to a rollout schedule. We changed too much too fast and tried for too many payoffs at once (in sales, client management, operations). That’s the number one reason the end-users got irritable and the reputation of the system was trashed almost immediately.

My personal project bible says much the same thing.
If you’re interested in this stuff at all, go read Rapid Development: Taming Wild Software Schedules, by Steve McConnell. It’s a terrific book; I’ve given about a dozen copies away to clients and team members.
OK, for the non-geeks in the crowd, what’s the relevance of this?
Traditional command-and-control government programs are like large software development projects; they are insanely complex, and to succeed, they require incredibly tight control over the participants. This both requires a huge ‘overhead’ of people doing the controlling, and a culture of tight control as well.
And like large software projects, they fail badly. The hysterisis…the time lag between action and output…is so large that it’s almost impossible to change or adapt. Small projects fail well; they have low hysterisis, and so when problems arise, they can adapt in response to real conditions.
As we’ve talked about race, one of my issues is not that change needs to happen; it’s that the breadth of change is so great that making it requires the kind of cumbersome, likely to fail or have unintended consequences programs that I’m discussing here. I’m trying to see a way through the problem that uses smaller more nimble, more effective programs.
Not only for dealing with issues of race, but for dealing with the array of issues that we confront as a society, and that, as a liberal, I’d like to use the power of society and the government to improve.


  1. Very insightful. It seems that in government, unless the project is huge, it doesn’t get votes (not enough pork for Senator Q from North Massajersey.) Since it is huge, it fails and results in unintended consequences.
    Of course, the only thing the government seems to be able to accomplish (in little, effective steps) is the eradication of our fundamental, Constitutionally protected rights.
    Strange, isn’t it?

  2. I wouldn’t say that government accomplishes nothing…
    …suffrage for women…
    …civil rights…
    …it just doesn’t do eveything we ask it to well, and we ask it to do too many things.

  3. A.l., huh???
    …railroads…In the USA at least, were mostly built by private enterprise, with a government subsidy consisting of land grants
    …airplanes…Designed and built mostly by private enterprise. The government’s main positive role in aviation was as a customer, sometimes the only customer willing to lay out big $$$ on things that kept flying into the ground.
    …satellites…might be a heck of a lot less expensive if launchers were privately built…
    …microelectronics…See airplanes – except the gov’t wasn’t that important of a customer until IC’s were quite well proven in civilian applications.
    …suffrage for women…Yep, the US gov’t corrected a blatant misjustice in it’s own laws, after a mere two centuries. Out in the marketplace, you don’t vote, you buy – and if women were unequal there, it was either the gov’ts fault (laws restricting their right to own and manage property), or it was because their earning capabilities were objectively lower back when most jobs required physical strength. (Less education was another factor, and must still be if you assume the differences in math and science achievement aren’t somehow related to actual differences in ability – and bad education has been the got’ts fault for quite a while, too.)
    …civil rights…OK, the gov’t actually did achieve something here beyond merely correcting unjust laws. It persuaded businessmen to follow their own self-interest in accepting all customers and hiring the best applicant. And then it screwed it up by requiring discrimination in favor of blacks (affirmative action), causing a resurgence of racism in the people discriminated against.

  4. mark…
    Oh, c’mon.
    Find me a significant (not feeder or spur) railroad that wasn’t funded by the real estate grants given by the gov’t. The successive waves of bonholders all invested based on the (incorrect) presumption that the land value would defend their bonds;
    Until the gov’t subsidised air travel, initially through postal routes, it was a curiosity;
    Yeah, now that there’s a technology base developed and paid for by the gov’t, it might make sense to privatize launch vehicles (except for the whole security issue);
    Excuse me?? Transistors and IC’s were developed to support government initiatives in computing other defense applications. Until those sunk costs were made, no commercial user would have subsidized that development.
    Suffrage?? It’s not like the population was united in clamoring for it over the objection of the lawyers and judges…
    I’m as wary of bureaucratic government as the average guy (who doesn’t live in Montana and worry about TEOTWAKI), but I do give it credit for a lot as well.
    When liberals acknowledge the problems, and when conservatives acknowledge the benefits, we’ll be making some progressin talking about politics in this country.

  5. And let’s not forget that a bunch of technology was developed at government expense because it was developed for the military first. I *love* my teflon cookware and raincoat, but before teflon was brought to me, the consumer, it was created for the military.

  6. “Find me a significant (not feeder or spur) railroad that wasn’t funded by the real estate grants given by the gov’t. The successive waves of bonholders all invested based on the (incorrect) presumption that the land value would defend their bonds”
    How about two? The Florida East Coast and Virginian were built almost entirely out of the pockets of Henry Flagler and Henry Rogers, respectively.
    The various federal land-grant railroads were in the western parts of the country (since the federal government didn’t have any free land in the east to grant), and even then, land sales didn’t come close to covering the initial costs of construction and operation.
    Railroad stocks and bonds before World War I were notorious for being a financial gamble: the payoff was great if the railroad proved successful, but many of them failed, wiped out the investors, reorganized with new funds, then failed again and wiped out more investors. It’s a miracle that any railroad at all got built in those days.

  7. One of the truly fascinating things about large-scale projects is that so much that was accomplished in the past (the national highway system, landing a man on the moon, etc.) seems to have gone so, well, smoothly compared to today’s projects (how long, again, did the 105 take and how much did it cost? What about Metrofail in downtown L.A.?). What has changed in project management from 30 years ago, or am I simply waxing nostalgic?
    I read somewhere that the term “software engineering” is an oxymoron. If it was engineering, it would be a repeatable process based upon unvarying laws like physics. It’s not — it’s an art form, a right-brained exercise in creativity. I think many business reengineering projects involving large-scale projects such as CRM and ERP fall into this category as well. That’s why those visionaries command those big salaries — one good project and you learn what the term “glass floor” means. ;)

  8. Henry, there is such a thing as software engineering. It just isn’t practiced in very many places. It requires a lot of self-discipline, and it probably costs a heck of a lot more than just hacking out something that does the job, if you don’t mind a few thousand bugs.
    It might be compared to electrical engineering a century ago – think Nikolai Tesla versus the methodical and boring engineers that actually made AC power systems practical. But it’s a heck of a lot more fun being Tesla than being Smith or whatever his name was…

  9. David … you may well have me, I need to do some more reading, I guess. I’ve read a few books on 19th century finance, they claimed that most of the money for the railroads (who typically went bankrupt more than once) was bonds and securities sold in Europe, and ‘secured’ by the land grants given the railroads.
    I’ll stand on my concept in general while acknowledging your point.

  10. Henry:
    One of the things I wonder about is simple…”when did we lose the ability to pour smooth concrete?”…the level of highway construction today is so much lower than what was built in the 60’s.

  11. mark:
    Yes, I agree that there is “software engineering”, but a) it exists primarily in DOD – level projects; and b) it gets into trouble because the tools and ‘materials’ being used are never mature.
    My dad built high-rise buildings; if one of his projects came in 10% over budget and 10% behind schedule, people got fired. People who can bring in tech projects in that range are heroes. Part of the reason is that building high-rises is a mature discipline; the tools, materials, and methodologies have been relatively stable for close to fifty years. When was the last time you built a piece of software where those were more than eighteen months old?

  12. Not only that, but the methodologies keep changing. I remember object-oriented programming which was going to save the world (heh, just dated myself) then it was the advent of the Capability Maturity Model, then Rational’s Unified Process and now I read about Agile and XP methods. The fact is, we’re all still trying things because most of it *doesn’t work*. Current methodologies fail to deliver a deterministic outcome. For the old argument of whether programming is art or science, art is winning. ;)

  13. That “pouring concrete” statement resonates as the crux of the modern project management problem — what happened to throw a monkey wrench into what used to appear simple? Why are bridges, high-rises, highways and mass transit consistently late, over budget or worse? Highways seem like an easy target. Check this:
    and this more recent example in OC:
    Highway 33 was, according to state documents, caused by “outdated information,” “failure to monitor the contractor’s performance” and “failure to terminate the contract immediately” when the contractor failed to perform.
    The 405/55 connector problem appears to be another “contractor oversight” issue. So are many of these problems a lack of disciplined project management, a lack of management period, or has something changed in the way contracts are handed out for projects that creates a perverse incentive to allow incompetence to continue?

  14. > My dad built high-rise buildings; if one of his projects came in 10% over budget and 10% behind schedule, people got fired. People who can bring in tech projects in that range are heroes.
    The folks doing the work usually get the blame for that difference. Why don’t the planners ever get blamed?
    One difference between tech project planning and building planning is that all of the IP creation occurs during building planning while a significant fraction of the tech IP is actually created during implementation.
    Also, the IP created can be significantly different in kind. Usually, the building IP is unrelated to deep functional issues. The building’s layout may have bad traffic patterns but the end result will probably be a building. A tech project planner may not have that kind of base to build from.

  15. “I’ve read a few books on 19th century finance, they claimed that most of the money for the railroads (who typically went bankrupt more than once) was bonds and securities sold in Europe, and ‘secured’ by the land grants given the railroads.”
    That was primarily true in the period between the end of the Civil War and the Panic of 1893, and the ‘European’ capital was primarily British. That ‘land grant security’ sounds to me like a bill of goods sold to Britons with more money than good sense, as the land along the completed sections was quickly sold off by the railroad to raise funds to continue construction.
    It was common throughout the 19th century for a prospective railroad to call for stock subscriptions from people who lived in the areas the railroad wished to build through, and especially before the Civil War for states and counties to finance railroad construction through stock subscription. The land grant program was designed to encourage railroad construction in unpopulated areas that otherwise would never have warranted any
    transportation improvements.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>