August VA systems outage crippled western hospitals, clinics
Day-long shutdown was longest of 14 recent service disruptions at Sacramento data processing center.
A day-long system outage at a new Veterans Affairs Department data processing center in northern California on Aug. 31 crippled critical information systems used to manage patient care at VA hospitals and clinics scattered across more than a third of the world, according to details from an internal VA after-action report.
The outage at the VA's Sacramento, Calif., regional center was the longest of 14 disruptions since that facility started hosting the suite of clinical applications that make up the Veterans Health Information System and Technology Architecture (VistA) earlier this year.
According to internal briefings, the Sacramento facility was created as part of a move by the VA to shift VistA computer operations from 126 local sites to four regional centers.
Since April, problems at the Sacramento center resulted in VistA downtime ranging from 15 minutes to the nine-hour outage on Aug. 31. That event knocked out vital information systems at hospitals and clinics operated by the Veterans Health Administration in Alaska, northern California, Los Angeles, Hawaii, Guam, Idaho, Nevada, Oregon, west Texas, American Samoa, the Philippines and Washington state.
The Sacramento failure first publicly surfaced at a hearing of the Senate Veterans Affairs Committee on Sept. 19, when Robert Howard, the VA's assistant secretary for information and technology, acknowledged it in response to a question by Sen. Patty Murray, D-Wash. Howard characterized the outage as a "big deal," but provided no details on its scope, scale or impact on patient care.
But Dr. Ben Davoren, director of clinical informatics at the San Francisco VA Medical Center, told a hearing of the House Veterans Affairs Committee last week that the failure on Aug. 31 was "the most significant technological threat to patient safety VA has ever had."
Dr. Bryan Volpp, associate chief of staff for clinical informatics at the VA's Northern California Healthcare System, told the House that the Aug. 31 outage all but sent VA hospitals and clinics in the western United States back to the paper age.
The outage, Volpp testified, forced medical staff to shift from the use of electronic medical records to writing notes and summaries on paper. Outpatient surgery was delayed because clinicians could not access forms, and doctors could not access electronic records for patients with scheduled appointments. Patients discharged that day could not be scheduled for follow-up appointments electronically.
Pharmacies at VA facilities connected to the Sacramento data center sputtered to a halt, Volpp said, because labeling and automatic dispensing equipment are controlled by VistA applications.
Paper records from Aug. 31 must be input into the electronic system by hand, Volpp said, a process that will take months.
Both Volpp and Davoren testified that the outage hit 17 VA medical facilities. But more than one VA medical staffer told Government Executive that this figure understates the scope of the outage, because the 17 are in turn electronically linked to numerous clinics and outpatient facilities.
A VA source in Hawaii said the Honolulu VA medical center's information systems were knocked out "because we use the Sacramento server, and Guam was knocked out because it goes through us." The San Francisco VA hospital, another source said, is electronically linked to multiple clinics in its area, as are hospitals and clinics in the region connected to the Sacramento data center.
While top VA information technology managers have touted the establishment of regional data centers as a way to eliminate downtime, insure continuity of operations and improve disaster recovery, Davoren told the House hearing that the Aug. 31 outage indicated to him that the regional model introduced a new single point of failure.
He testified that in case of an outage, the Sacramento data center was supposed to "failover" to another regional center in Denver, but did not. The after-action report did not address why this switchover did not happen. Volpp testified that another backup system, a read-only backup of patient data, was unavailable on Aug. 31 due to work by the Sacramento center to recreate accounts holding the data.
The VA's plans to establish four regional data centers are part of an overall effort to centralize IT resources and personnel to help eliminate the computer security breaches that have plagued the VA over the past year.
But Davoren told the House hearing that medical center employees expressed concerns as early as 2005 that "the regionalization of IT resources would create new points of failure that could not be controlled by the sites experiencing the impact, and that the system redundancy required to prevent this was never listed as a prerequisite to centralization of critical patient care IT resources."
The VA did not immediately respond to queries from Government Executive about the outages in Sacramento or how it intends to remedy the situation. Howard, the VA IT director, told the Senate VA hearing on Sept. 19 that the department intends to add "more robust backup capability" to help mitigate system downtime at the regional data centers.
Howard added that his staff is examining whether or not there is something about the VistA software itself -- developed over years and hosted at the local medical facility level -- that does not lend itself to hosting at a regional data center. Until that process is completed, Howard said the VA will cease any further migration of VistA applications to regional data centers.