August VA systems outage crippled western hospitals, clinics

A day-long system outage at a new Veterans Affairs Department data processing center in northern California on Aug. 31 crippled critical information systems used to manage patient care at VA hospitals and clinics scattered across more than a third of the world, according to details from an internal VA after-action report.

The outage at the VA's Sacramento, Calif., regional center was the longest of 14 disruptions since that facility started hosting the suite of clinical applications that make up the Veterans Health Information System and Technology Architecture (VistA) earlier this year.

According to internal briefings, the Sacramento facility was created as part of a move by the VA to shift VistA computer operations from 126 local sites to four regional centers.

Since April, problems at the Sacramento center resulted in VistA downtime ranging from 15 minutes to the nine-hour outage on Aug. 31. That event knocked out vital information systems at hospitals and clinics operated by the Veterans Health Administration in Alaska, northern California, Los Angeles, Hawaii, Guam, Idaho, Nevada, Oregon, west Texas, American Samoa, the Philippines and Washington state.

The Sacramento failure first publicly surfaced at a hearing of the Senate Veterans Affairs Committee on Sept. 19, when Robert Howard, the VA's assistant secretary for information and technology, acknowledged it in response to a question by Sen. Patty Murray, D-Wash. Howard characterized the outage as a "big deal," but provided no details on its scope, scale or impact on patient care.

But Dr. Ben Davoren, director of clinical informatics at the San Francisco VA Medical Center, told a hearing of the House Veterans Affairs Committee last week that the failure on Aug. 31 was "the most significant technological threat to patient safety VA has ever had."

Dr. Bryan Volpp, associate chief of staff for clinical informatics at the VA's Northern California Healthcare System, told the House that the Aug. 31 outage all but sent VA hospitals and clinics in the western United States back to the paper age.

The outage, Volpp testified, forced medical staff to shift from the use of electronic medical records to writing notes and summaries on paper. Outpatient surgery was delayed because clinicians could not access forms, and doctors could not access electronic records for patients with scheduled appointments. Patients discharged that day could not be scheduled for follow-up appointments electronically.

Pharmacies at VA facilities connected to the Sacramento data center sputtered to a halt, Volpp said, because labeling and automatic dispensing equipment are controlled by VistA applications.

Paper records from Aug. 31 must be input into the electronic system by hand, Volpp said, a process that will take months.

Both Volpp and Davoren testified that the outage hit 17 VA medical facilities. But more than one VA medical staffer told Government Executive that this figure understates the scope of the outage, because the 17 are in turn electronically linked to numerous clinics and outpatient facilities.

A VA source in Hawaii said the Honolulu VA medical center's information systems were knocked out "because we use the Sacramento server, and Guam was knocked out because it goes through us." The San Francisco VA hospital, another source said, is electronically linked to multiple clinics in its area, as are hospitals and clinics in the region connected to the Sacramento data center.

While top VA information technology managers have touted the establishment of regional data centers as a way to eliminate downtime, insure continuity of operations and improve disaster recovery, Davoren told the House hearing that the Aug. 31 outage indicated to him that the regional model introduced a new single point of failure.

He testified that in case of an outage, the Sacramento data center was supposed to "failover" to another regional center in Denver, but did not. The after-action report did not address why this switchover did not happen. Volpp testified that another backup system, a read-only backup of patient data, was unavailable on Aug. 31 due to work by the Sacramento center to recreate accounts holding the data.

The VA's plans to establish four regional data centers are part of an overall effort to centralize IT resources and personnel to help eliminate the computer security breaches that have plagued the VA over the past year.

But Davoren told the House hearing that medical center employees expressed concerns as early as 2005 that "the regionalization of IT resources would create new points of failure that could not be controlled by the sites experiencing the impact, and that the system redundancy required to prevent this was never listed as a prerequisite to centralization of critical patient care IT resources."

The VA did not immediately respond to queries from Government Executive about the outages in Sacramento or how it intends to remedy the situation. Howard, the VA IT director, told the Senate VA hearing on Sept. 19 that the department intends to add "more robust backup capability" to help mitigate system downtime at the regional data centers.

Howard added that his staff is examining whether or not there is something about the VistA software itself -- developed over years and hosted at the local medical facility level -- that does not lend itself to hosting at a regional data center. Until that process is completed, Howard said the VA will cease any further migration of VistA applications to regional data centers.

Stay up-to-date with federal news alerts and analysis — Sign up for GovExec's email newsletters.
Close [ x ] More from GovExec

Thank you for subscribing to newsletters from
We think these reports might interest you:

  • Sponsored by G Suite

    Cross-Agency Teamwork, Anytime and Anywhere

    Dan McCrae, director of IT service delivery division, National Oceanic and Atmospheric Administration (NOAA)

  • Data-Centric Security vs. Database-Level Security

    Database-level encryption had its origins in the 1990s and early 2000s in response to very basic risks which largely revolved around the theft of servers, backup tapes and other physical-layer assets. As noted in Verizon’s 2014, Data Breach Investigations Report (DBIR)1, threats today are far more advanced and dangerous.

  • Sponsored by One Identity

    One Nation Under Guard: Securing User Identities Across State and Local Government

    In 2016, the government can expect even more sophisticated threats on the horizon, making it all the more imperative that agencies enforce proper identity and access management (IAM) practices. In order to better measure the current state of IAM at the state and local level, Government Business Council (GBC) conducted an in-depth research study of state and local employees.

  • Sponsored by Aquilent

    The Next Federal Evolution of Cloud

    This GBC report explains the evolution of cloud computing in federal government, and provides an outlook for the future of the cloud in government IT.

  • Sponsored by LTC Partners, administrators of the Federal Long Term Care Insurance Program

    Approaching the Brink of Federal Retirement

    Approximately 10,000 baby boomers are reaching retirement age per day, and a growing number of federal employees are preparing themselves for the next chapter of their lives. Learn how to tackle the challenges that today's workforce faces in laying the groundwork for a smooth and secure retirement.

  • Sponsored by Hewlett Packard Enterprise

    Cyber Defense 101: Arming the Next Generation of Government Employees

    Read this issue brief to learn about the sector's most potent challenges in the new cyber landscape and how government organizations are building a robust, threat-aware infrastructure

  • Sponsored by Aquilent

    GBC Issue Brief: Cultivating Digital Services in the Federal Landscape

    Read this GBC issue brief to learn more about the current state of digital services in the government, and how key players are pushing enhancements towards a user-centric approach.


When you download a report, your information may be shared with the underwriters of that document.