GovernmentExecutive.com

Complexity is innate to software, and complex things fail. Complex systems interact in unexpected ways. The more they're asked to do and the larger they get, the more likely unanticipated consequences will pop up.

The question isn't whether or not software projects will sometimes fail. They will. A conservative estimate of the cost of federal software failures pegs the annual value at $3 billion, according to Robert Charette, president of ITABHI Corp., a Spotsylvania, Va., risk-management consultancy.

Take the failure of an Internal Revenue Service electronic fraud detection system, for example. The IRS worked for five years to replace its old system with new Web-based software. Was the failure of the Web-based software a function of its essence, or was it an accident?

The tax agency had assurances from contractor Computer Sciences Corp. of El Segundo, Calif., that the new system would be online by January 2006. It wasn't. The IRS had no contingency plan and already had shut down its original system, leaving the 2006 tax season wide open for cheats.

As a result, according to a Treasury Department inspector general for tax administration report (TIGTA 2006-20-108), tax dodgers may have gotten away with $318.3 million in improper refunds. The IRS had invested $20.5 million in the Web-based system through last April.

When auditors investigated, they found that developers were unfamiliar with user needs, teams failed to coordinate with each other (so when one team made a change affecting others, the other teams wouldn't know about it, requiring a software fix), there was excessive turnover among CSC and IRS employees, and CSC repeatedly missed intermediate deadlines while still promising to complete the project on time.

"If anything could go wrong in a system-development-type atmosphere, it went wrong," says Margaret E. Begg, a Treasury tax administration assistant IG.

What happened at the IRS was neither an accident nor an example of software's complex essence, says Charette. It "doesn't even deserve the name 'failure,' " he says. "It's a blunder."

No governance process was in place to resolve difficulties and, despite a critical need for the software to perform, risks surrounding the project piled up mostly unattended, the IG found. When, before the project's collapse, the IRS director of refund crimes requested an independent study to scrutinize whether CSC would be able to deliver on its promises, he was informed that "the time needed to resolve 'a few existing glitches' " would prevent the study from starting until after the final deadline.

At another point, when an IRS employee found that CSC grossly exaggerated the work it had completed, the IRS "did not pursue the matter," IG investigators found.

CSC refused to comment, citing contractual obligations. Asked about specific problems cited by auditors, the IRS responded with a statement that it is "taking a number of steps to ensure the continued integrity of this system" and has revamped its governance structure so "significant risks and issues are identified early on and elevated to the appropriate executive level for attention."

A blunder is classic and boring, the stuff of basic project management, Charette says. It's made up of avoidable execution mistakes. Proper management of a software project includes building into the design enough wiggle room to absorb a reasonable number of mistakes.

Blunders are avoidable, but also unlikely to disappear. Institutional pressures conspire to ensure that, despite having blundered many times, the federal government likely will blunder again. Getting a project funded often means overpromising results while downplaying costs.

Managers have an incentive to make their projects look good, no matter their true status, often in the hope that problems will clear up before anybody else can notice. And, of course, deception is self-reinforcing and self-perpetuating; once it starts, it spreads. The problem is systemic: Industry proposes unrealistic costs and understates risks, while government agencies blithely accept them.

"CSC led them to believe that [the fraud system] would be operational, right up until the end, when in fact, they were unable to deliver," notes Begg.

In August 2005, IRS managers began to realize they had a serious problem. The agency started holding biweekly meetings with contractors, and in early December, executives started meeting daily. During the project's last couple of months, technical staffers were sending out two or three daily voicemail updates.

Finally, on April 19, 2006, the IRS decided it had had enough. The April deadline for most Americans to file their personal tax returns had come and gone with no electronic fraud detection system in place. The agency halted further development work and said it would restore the old system, the one that in 2001 it had said needed replacement. IRS says the old system will be working in time for the 2007 tax season.

COMMENTS

• I believe this problem is played out throughout the government community. I recall a postal system for sorting that was to replace the old manual process. Unfortunately, USPS adjusted their staffing for the new system (fewer staff needed to sort). Unfortunately, the new system went in just before Christmas (heavy load) and it failed. When will the federal government understand the need for clear, precise requirements and management of requirement changes? The rush to deploy seems to mean eliminating this important step. As individuals, we would never spend our own money without defining what we wanted. Would you go to the builder and tell him that you wanted him to build you a house, providing only a vague description of what you wanted? GovExec.com reader Posted March 23, 2007 9:06 AM
• Twenty million? That is chump change! How much has been spent on DIMHRS to this point, 20 million per year for the last 10 years? And what do we have to show for it? The marines refuse it, the navy does not want it, and the air force and army are trying to figure out how they can use it. All the while, the old faulty legacy systems keep running on shoestring budgets, while the generals and admirals and SESers tout the need to go to new technology to save the immense costs and inequities of the old systems that in reality, are working just fine, thank you. Balderdash! GovExec.com reader Posted March 22, 2007 12:32 PM
• Lack of competence in contract development and oversight is broader than IT-related acquisitions. Is there anyone who hasn't heard the old saw, "I'll know it when I see it?" Use of this approach, historically applied to internal staffs, alleviates managers and others from having to perform "heavy thinking" about their true needs, allowing our petty potentates the bureaucratic equivalent of a window-shopping stroll through the local mall. The ugly underbelly of that concept, when applied in the context of an acquisition, is that we (managers and others) are often unable or unwilling to adequately define and articulate just what it is we want, leaving delivery variables up to the contractor's discretion. Then, when the product or service doesn't meet our later expectations, we of course blame it on the contractor. When we ill-define our needs and then choose the least expensive vendor to provide a service or product, it's not surprising that the winning bidder's cost structure won't support "fishing expeditions" or multiple "spec solutions" for us to mull over. There are contractors who perform disreputably, and others who legitimately operate within the wiggle room provided by unspecific work orders. If we did a better job of holding up our end of the acquisition workload, we'd have better luck with contractors. GovExec.com reader Posted March 22, 2007 8:02 AM