BalkansCat/Shutterstock.com

The U.S. Spent $2.2 Million on a Cybersecurity System That Wasn’t Implemented — and Might Have Stopped a Major Hack

The software company SolarWinds unwittingly allowed hackers’ code into thousands of federal computers. A cybersecurity system called in-toto, which the government paid to develop but never required, might have protected against this.

As America struggles to assess the damage from the devastating SolarWinds cyberattack discovered in December, ProPublica has learned of a promising defense that could shore up the vulnerability the hackers exploited: a system the federal government funded but has never required its vendors to use.

The massive breach, which U.S. intelligence agencies say was “likely Russian in origin,” penetrated the computer systems of critical federal agencies, including the Department of Homeland Security, the Treasury Department, the National Institutes of Health and the Department of Justice, as well as a number of Fortune 500 corporations. The hackers remained undetected, free to forage, for months.

The hackers infiltrated the systems by inserting malware into routine software updates that SolarWinds sent to customers to install on its products, which are used to monitor internal computer networks. Software updates customarily add new features, remove bugs and boost security. But in this instance, the hackers commandeered the process by slipping in malicious code, creating secret portals (called “back doors”) that granted them access to an untold bounty of government and company secrets.

The incursion became the latest — and, it appears, by far the worst — in a string of hacks targeting the software supply chain. Cybersecurity experts have voiced concern for years that existing defenses, which focus on attacks against individual end users, fail to spot malware planted in downloads from trusted software suppliers. Such attacks are especially worrisome because of their ability to rapidly distribute malicious computer code to tens of thousands of unwitting customers.

This problem spurred development of a new approach, backed by $2.2 million in federal grants and available for free, aimed at providing end-to-end protection for the entire software supply pipeline. Named in-toto (Latin for “as a whole”), it is the work of a team of academics led by Justin Cappos, an associate computer science and engineering professor at New York University. Cappos, 43, has made securing the software supply chain his life’s work. In 2013, Popular Science named him as one of its “Brilliant Ten” scientists under 40.

Cappos and his colleagues believe that the in-toto system, if widely deployed, could have blocked or minimized the damage from the SolarWinds attack. But that didn’t happen: The federal government has taken no steps to require its software vendors, such as SolarWinds, to adopt it. Indeed, no government agency has even inquired about it, according to Cappos.

“In security, you almost never go from making something possible to impossible,” Cappos told ProPublica, during two video interviews from Shanghai, where he is teaching. “You go from making it easy to making it hard. We would have made it much harder for the [SolarWinds] attackers, and most likely would have stopped the attack.” Although the SolarWinds breach was a “really sneaky” approach, Cappos said, “in-toto definitely can protect against this. It’s very possible to catch it.”

In-toto’s system has supporters among experts in the government and corporations. When ProPublica asked Robert Beverly, who oversees in-toto’s federal grant as a program director at the National Science Foundation, whether using in-toto could have saved the government from the hack, he replied, “Absolutely. There seems to be some strong evidence that had some of the, or all of the, in-toto technologies been in place, this would have been mitigated to some extent.” Beverly, whose NSF responsibilities include “cybersecurity innovation for cyberinfrastructure” and who is on leave from his post as a computer science professor at the Naval Postgraduate School, added that it’s impossible to know for sure what impact in-toto would have had, and that the system remains at an early stage of adoption. “Unfortunately,” said Beverly, “it often takes some of these kinds of events to convince people to use these kinds of technologies.”

Some companies have embraced in-toto, and others, like Microsoft, have expressed interest. “I am a big fan of in-toto,” Kay Williams, head of Microsoft’s initiatives in open source and supply-chain security, said in an email to ProPublica. A second Microsoft program manager, Ralph Squillace, praised in-toto in a recent NYU press release for applying “precisely to the problems of supply chain confidence the community expects distributed applications to have in the real world.” (After Williams’ initial response, Microsoft declined to comment further.)

One senator blasted the government’s failure to use a system it paid for. “The U.S. government invested millions of dollars in developing technology that can protect against this threat, and while several large technology companies have already adopted it, they are the exception,” said Sen. Ron Wyden, D-Ore., a member of the Senate Intelligence Committee. “The government can speed up industry adoption of this best practice by requiring every government contractor to implement the best available technology to protect their supply chains.”

The in-toto system requires software vendors to map out their process for assembling computer code that will be sent to customers, and it records what’s done at each step along the way. It then verifies electronically that no hacker has inserted something in between steps. Immediately before installation, a pre-installed tool automatically runs a final check to make sure that what the customer received matches the final product the software vendor generated for delivery, confirming that it wasn’t tampered with in transit.

Cappos and a team of colleagues have worked to develop the in-toto approach for years. It’s been up and running since 2018. The project received a three-year grant from the National Science Foundation that year, aimed at promoting “widespread practical use” of in-toto. (Later in 2018, President Donald Trump signed the Federal Acquisition Supply Chain Security Act, aimed at protecting government secrets from software supply-chain threats.)

In-toto could block and reveal countless cyberattacks that currently go undetected, according to Cappos, whose team includes Santiago Torres-Arias, an assistant electrical and computer engineering professor at Purdue University, and Reza Curtmola, co-director of the New Jersey Institute of Technology’s Cybersecurity Research Center. In an August 2019 paper and presentation to the USENIX computer conference, titled “in-toto: Providing farm-to-table guarantees for bits and bytes,” Cappos’ team reported studying 30 major supply-chain breaches dating back to 2010. In-toto, they concluded, would have prevented between 83% and 100% of those attacks.

“It’s available to everyone for free, paid for by the government, and should be used by everyone,” said Cappos. “People may still be able to break in and try to hack around it. But this is a necessary first step and will catch a ton of these things.” The slow pace of adoption is “really disappointing,” Cappos added. “In the long game, we’ll win. I just don’t know that we want to go through the pain that it’ll take for everyone to wise up.”

One of in-toto’s earliest adopters, starting in 2018, was Datadog, a SolarWinds competitor that provides monitoring software for internet cloud applications. Now a publicly traded company with 2020 revenues of nearly $600 million, its customers include Nasdaq, Whole Foods and Samsung. Datadog uses in-toto to protect the security of its software updates. In an NYU press release, Datadog staff security engineer Trishank Kuppusamy, who worked on the program’s design and implementation, said that what distinguishes in-toto is that it “has been designed against a very strong threat model that includes nation-state attackers.” (Datadog did not reply to ProPublica’s requests for comment.)

The General Services Administration, which provides access to software for federal government agencies, still lists SolarWinds products available for purchase. But it said in a statement that “compromised versions” of SolarWinds programs identified by DHS are no longer available.

SolarWinds itself declined to weigh in on whether its hack could have been prevented. “We are not going to speculate on in-toto and its capabilities,” a spokesman said in an emailed statement. “We are focused on protecting our customers, hardening our security and collaborating with the industry to understand the attack and prevent similar attacks in the future.”

Previously little known to the general public, SolarWinds is a public company based in Austin, Texas, with projected 2020 revenues of just over $1 billion. It boasts of providing software to 320,000 customers in 199 countries, including 499 of the Fortune 500 companies. In a recent SEC filing, the company said its flagship Orion products, the vehicle for the cyberattack, provide about 45% of its revenues. A SolarWinds slogan: “We make IT look easy.”

After the hack was discovered, SolarWinds’ stock plunged, and it is now facing shareholder lawsuits. The company has shifted aggressively into damage-control mode, hiring CrowdStrike, a top cybersecurity firm; elite Washington lobbyists; a crisis-communications advisor; and the newly formed consulting team of Christopher Krebs, the former director of the Cybersecurity and Infrastructure Security Agency (who was famously fired for contradicting Trump’s claims of mass voting fraud) and Alex Stamos, former security chief at Facebook.

News of what’s now known as the SolarWinds attack first came on Dec. 8. That’s when FireEye, perhaps the nation’s preeminent hack-hunter, announced that it had itself fallen victim to a “highly sophisticated state-sponsored adversary” that had broken into its servers and stolen its “Red Team tools,” which FireEye uses to try to hack into the computer networks of its clients as a test of their cyber-defenses. FireEye soon discovered the attackers had gained access through corrupted updates to the SolarWinds Orion network-monitoring software that it used.

On the evening of Dec. 13, CISA issued an emergency directive, identifying SolarWinds as ground zero for the hack and alerting federal agencies using Orion products to disconnect them immediately. Over the following weeks, investigators discovered that SolarWinds had been targeted back in early September 2019, when hackers started testing their ability to inject code into its software updates. After remaining undetected for months, they inserted malware in new updates between February and June 2020. SolarWinds estimated these infected updates affected “fewer than 18,000 of its customers.”

Precisely what the hackers saw, and stole, has yet to be determined and is under investigation. But the full impact of the breach is becoming clearer, as we now know it touches several tech companies, including Microsoft. The software giant has also labored to limit the damage by helping seize an internet domain in the U.S. that the hackers used to siphon data from some SolarWinds customers.

Stamos told the Financial Times, in an interview after being hired to help SolarWinds, that he believed the attackers had embedded hidden code that would continue to give them access to companies and government agencies for years. He compared the situation to Belgian and French farmers going out into their fields where two world wars were fought and discovering an “iron harvest” of unexploded ordnance each spring.

Dmitri Alperovitch, who co-founded CrowdStrike (the cybersecurity firm SolarWinds has hired to investigate the hack) before leaving last year to start a nonprofit policy group, said he thinks that, in theory, the in-toto system could work. But he warned that software is so complex, with many products and companies in the supply chain, that no one defense is a panacea. Still, he agrees that in-toto could provide protection, and said “it’s always a good thing to have more protection for supply chains.”

Russian intelligence services have clearly identified supply-chain attacks “as a much better way to get in,” offering “a much bigger set of targets,” Alperovitch said. “This is an indictment of the entire cybersecurity industry, as well as the intelligence community, that they were able to orchestrate such a broad, sweeping attack right under our noses.”

This article was originally published in ProPublica. It has been republished under the Creative Commons license.  ProPublica is a Pulitzer Prize-winning investigative newsroom. Sign up for their newsletter.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.