Unplugging Data Centers

When state officials decided to merge Oregon's mish-mash of 12 agency data centers into one secure, cutting-edge facility in 2005, the task before them seemed easy enough. But three years and $63 million into the project, the state auditor was howling that Oregonians' private information was no more secure than before and agencies were complaining they were shelling out more in user fees than they'd spent to maintain their own facilities.

It's taken another three years for the consolidation project to start bearing fruit. Now Oregon is paying less per unit on all its major data center components than it did in 2007. The shared center is projecting total savings of about $200 million by 2013 over what agencies would have paid individually, Deputy Administrator Kurtis Danka and his boss, Administrator Julie Bozzi, told the state legislature in April.

As the federal government embarks on a massive program to consolidate its roughly 2,100 data centers down to 1,300 or fewer by 2015, federal officials from the Energy Department and elsewhere have come to Salem to tour Oregon's new facility and to learn from the state's experience, Danka says.

His advice: Put as much work into security on the front end as possible; accept upfront that consolidation will get messy, requiring bits and pieces of server space for data that just can't be brought onto a standardized platform; and, perhaps most important, don't necessarily believe what officials tell you about the amount of storage space they'll need.

Oregon officials began their consolidation effort, for instance, figuring they would need about a 40-terabyte capacity, Danka says. The central facility will soon be managing 800 terabytes of information.

Because data storage often is integrated into agencies' operations and budgets, the people overseeing it typically have a poor sense, and even poorer set of records, of how much information they're keeping and how much they're paying to house it, he says.

"Another lesson learned is that you really have to establish a good forecasting mechanism" for future usage, Danka says, "because they never believe they're going to use as much as they do."

Data centers are essentially rooms filled with computer servers that store everything from email and payroll systems to confidential satellite imagery and weapons plans.

The number of federal facilities has more than quadrupled as government's reliance on technology has grown, with centers increasing from 432 in 1998 to 2,093 in 2010.

That massive growth came at a price, as the energy and staff necessary for maintaining thousands of different infrastructures and for upgrading diverse operating systems ballooned. Federal Chief Information Officer Vivek Kundra has estimated consolidating the more than 2,000 centers will save the government $3 billion in rent, upkeep and operations over the next four years. Some private studies have put the savings much higher.

The CIO's office has announced it plans to shutter 137 centers in 2011, but because the facilities can range from the size of a small apartment to that of a warehouse, it's difficult to tell what immediate effect those closures will have on federal spending. A Government Accountability Office report due out in mid-July will analyze each agency's planned closures and likely give a better sense of the scope of this round.

Kundra expects to save an additional $5 billion annually by moving about one-fourth of the government's information technology expenditures to the cloud-even larger server banks where agencies can buy data storage as they would a utility, paying only for the space they use.

Brad Ellison, data center architect at Intel Corp., managed the chip-maker's consolidation of 147 data centers in 2006 to 91 today, about the same proportional reduction the government is attempting. Intel planned to slash its way down to six centers companywide but that goal quickly proved impractical, he says.

"A data center strategy is not a static thing," Ellison says. "It's a living document that has to be adjusted as the realities of the business evolve."

In the government's case, he adds, consolidation could be stymied by the sheer diversity of agency missions.

Data centers operate best when they're running standard programs, he notes. Programs that pop up at different times can require all sorts of unique software and other technology that's often written into their authorizing legislation.

Security requirements, also often set in law, complicate the situation even more.

The Internal Revenue Service, for instance, mandates that hard drives containing taxpayer information, including those at state tax departments that report data to the IRS, must be physically destroyed if they get corrupted.

Oregon's center stores most of its data on service area networks, which save space by parceling out bits of information across a dozen or more hard drives, Danka says. The center's operators can't risk destroying 20 hard drives because of one error, so they must segregate a handful of servers for IRS data.

Data storage consolidation is ultimately about relationships and communication. As the government moves on from low-hanging fruit to more vital operations, Danka says, agency chief information officers will have to spend a lot of time convincing operational staffers their data is in safe hands.

"This wasn't at all a 'build it and they will come,' " he says of Oregon's initiative. "It was a governor's directive . . . and that created a rocky start for the agencies. Establishing those relationships with the customers is very important. Given the amount of consolidation [and] the aggressiveness of the schedule, our success depended on them wanting to be our partners."