August VA systems outage crippled western hospitals, clinics

A day-long system outage at a new Veterans Affairs Department data processing center in northern California on Aug. 31 crippled critical information systems used to manage patient care at VA hospitals and clinics scattered across more than a third of the world, according to details from an internal VA after-action report.

The outage at the VA's Sacramento, Calif., regional center was the longest of 14 disruptions since that facility started hosting the suite of clinical applications that make up the Veterans Health Information System and Technology Architecture (VistA) earlier this year.

According to internal briefings, the Sacramento facility was created as part of a move by the VA to shift VistA computer operations from 126 local sites to four regional centers.

Since April, problems at the Sacramento center resulted in VistA downtime ranging from 15 minutes to the nine-hour outage on Aug. 31. That event knocked out vital information systems at hospitals and clinics operated by the Veterans Health Administration in Alaska, northern California, Los Angeles, Hawaii, Guam, Idaho, Nevada, Oregon, west Texas, American Samoa, the Philippines and Washington state.

The Sacramento failure first publicly surfaced at a hearing of the Senate Veterans Affairs Committee on Sept. 19, when Robert Howard, the VA's assistant secretary for information and technology, acknowledged it in response to a question by Sen. Patty Murray, D-Wash. Howard characterized the outage as a "big deal," but provided no details on its scope, scale or impact on patient care.

But Dr. Ben Davoren, director of clinical informatics at the San Francisco VA Medical Center, told a hearing of the House Veterans Affairs Committee last week that the failure on Aug. 31 was "the most significant technological threat to patient safety VA has ever had."

Dr. Bryan Volpp, associate chief of staff for clinical informatics at the VA's Northern California Healthcare System, told the House that the Aug. 31 outage all but sent VA hospitals and clinics in the western United States back to the paper age.

The outage, Volpp testified, forced medical staff to shift from the use of electronic medical records to writing notes and summaries on paper. Outpatient surgery was delayed because clinicians could not access forms, and doctors could not access electronic records for patients with scheduled appointments. Patients discharged that day could not be scheduled for follow-up appointments electronically.

Pharmacies at VA facilities connected to the Sacramento data center sputtered to a halt, Volpp said, because labeling and automatic dispensing equipment are controlled by VistA applications.

Paper records from Aug. 31 must be input into the electronic system by hand, Volpp said, a process that will take months.

Both Volpp and Davoren testified that the outage hit 17 VA medical facilities. But more than one VA medical staffer told Government Executive that this figure understates the scope of the outage, because the 17 are in turn electronically linked to numerous clinics and outpatient facilities.

A VA source in Hawaii said the Honolulu VA medical center's information systems were knocked out "because we use the Sacramento server, and Guam was knocked out because it goes through us." The San Francisco VA hospital, another source said, is electronically linked to multiple clinics in its area, as are hospitals and clinics in the region connected to the Sacramento data center.

While top VA information technology managers have touted the establishment of regional data centers as a way to eliminate downtime, insure continuity of operations and improve disaster recovery, Davoren told the House hearing that the Aug. 31 outage indicated to him that the regional model introduced a new single point of failure.

He testified that in case of an outage, the Sacramento data center was supposed to "failover" to another regional center in Denver, but did not. The after-action report did not address why this switchover did not happen. Volpp testified that another backup system, a read-only backup of patient data, was unavailable on Aug. 31 due to work by the Sacramento center to recreate accounts holding the data.

The VA's plans to establish four regional data centers are part of an overall effort to centralize IT resources and personnel to help eliminate the computer security breaches that have plagued the VA over the past year.

But Davoren told the House hearing that medical center employees expressed concerns as early as 2005 that "the regionalization of IT resources would create new points of failure that could not be controlled by the sites experiencing the impact, and that the system redundancy required to prevent this was never listed as a prerequisite to centralization of critical patient care IT resources."

The VA did not immediately respond to queries from Government Executive about the outages in Sacramento or how it intends to remedy the situation. Howard, the VA IT director, told the Senate VA hearing on Sept. 19 that the department intends to add "more robust backup capability" to help mitigate system downtime at the regional data centers.

Howard added that his staff is examining whether or not there is something about the VistA software itself -- developed over years and hosted at the local medical facility level -- that does not lend itself to hosting at a regional data center. Until that process is completed, Howard said the VA will cease any further migration of VistA applications to regional data centers.

Stay up-to-date with federal news alerts and analysis — Sign up for GovExec's email newsletters.
Close [ x ] More from GovExec

Thank you for subscribing to newsletters from
We think these reports might interest you:

  • Going Agile:Revolutionizing Federal Digital Services Delivery

    Here’s one indication that times have changed: Harriet Tubman is going to be the next face of the twenty dollar bill. Another sign of change? The way in which the federal government arrived at that decision.

  • Cyber Risk Report: Cybercrime Trends from 2016

    In our first half 2016 cyber trends report, SurfWatch Labs threat intelligence analysts noted one key theme – the interconnected nature of cybercrime – and the second half of the year saw organizations continuing to struggle with that reality. The number of potential cyber threats, the pool of already compromised information, and the ease of finding increasingly sophisticated cybercriminal tools continued to snowball throughout the year.

  • Featured Content from RSA Conference: Dissed by NIST

    Learn more about the latest draft of the U.S. National Institute of Standards and Technology guidance document on authentication and lifecycle management.

  • GBC Issue Brief: The Future of 9-1-1

    A Look Into the Next Generation of Emergency Services

  • GBC Survey Report: Securing the Perimeters

    A candid survey on cybersecurity in state and local governments

  • The New IP: Moving Government Agencies Toward the Network of The Future

    Federal IT managers are looking to modernize legacy network infrastructures that are taxed by growing demands from mobile devices, video, vast amounts of data, and more. This issue brief discusses the federal government network landscape, as well as market, financial force drivers for network modernization.

  • eBook: State & Local Cybersecurity

    CenturyLink is committed to helping state and local governments meet their cybersecurity challenges. Towards that end, CenturyLink commissioned a study from the Government Business Council that looked at the perceptions, attitudes and experiences of state and local leaders around the cybersecurity issue. The results were surprising in a number of ways. Learn more about their findings and the ways in which state and local governments can combat cybersecurity threats with this eBook.


When you download a report, your information may be shared with the underwriters of that document.