The Surveillance State: How We Got Here


On December 20, 2002, a Senate Intelligence Committee that included Sen. Ron Wyden, D-Ore., today one of the most vociferous critics of the so-called "surveillance state," came to the following conclusion in its official report on the mistakes that led to 9/11: The National Security Agency had harmed U.S. counterterrorism efforts that might have prevented that terrible day because of the agency's "failure to address modern communications technology aggressively."

The report, a joint effort of the Senate committee and the House Permanent Select Committee on Intelligence, blamed "NSA's cautious approach to any collection of intelligence relating to activities in the United States, and insufficient collaboration between NSA and the FBI regarding the potential for terrorist attacks within the United States."

The Senate-House report said the NSA simply could not keep up with the explosion of information technology. "Only a tiny fraction" of the NSA's 650 million daily intercepts worldwide "are actually ever reviewed by humans, and much of what is collected gets lost in the deluge of data," the report said. In interviews at the time, then-NSA Director Michael Hayden explained why: The NSA, originally authorized to conduct monitoring only overseas, was effectively a Cold War dinosaur that was going "deaf" since its main mission of tracking "signals intelligence," known as Sigint, from the Soviet Union had ended.

"We have gone from chasing the telecommunications structure of a slow-moving, technologically inferior, resource-poor nation-state—and we could do that pretty well – to chasing a communications structure in which an al-Qaida member can go into a storefront in Istanbul and buy for $100 a communications device that is absolutely cutting edge, and for which he has had to make no investment in its development. That's what we've got to deal with," Hayden told me in an interview in mid-2002.

In congressional testimony leading up to that critical Senate-House report, Hayden explained the NSA had gone from tracking a relatively small number of Soviet communications pipelines — microwave transmissions, for example, from Moscow to various ICBM bases — to trying to keep up with billions of conversations on phones and emails in a world in which technological borders had been erased, and much of this traffic was now being routed through the United States. This huge new challenge was coming at a time when the super-secret agency had "downsized about a third of its manpower and about the same proportion of its budget in the '90s," the era of the so-called post-Cold War peace dividend, Hayden said in his testimony. "That's the same decade when mobile cell phones increased from 16 million to 741 million—an increase of 50 times. That's the same decade when Internet users went from about four to 361 million."

These perceived deficiencies, and the NSA's aggressive efforts to redress them since then, make up the real backdrop to the latest scandal that has engulfed Washington, this time over what appears to be a massive infringement of American civil liberties. And despite the outrage voiced by senators such as Wyden and other critics, the truth about what the NSA and intelligence and investigative community is doing is far more complex than the rhetoric might lead you to believe.

Most important of all, almost the entire U.S. government has been on board in promoting it.

After many struggles and failures in the last decade, the NSA did finally come up with new approaches to keeping up with the traffic. One such approach was the NSA's "PRISM" program, disclosed Thursday by The Washington Post and the Guardian newspaper. The newspapers revealed that the NSA and FBI have set up a program to tap directly into the central servers of nine leading U.S. Internet companies, "extracting audio and video chats, photographs, e-mails, documents, and connection logs that enable analysts to track one target or trace a whole network of associates," as the Post wrote. The program was reportedly set up in cooperation with the major companies, including Microsoft, Yahoo, Google, Facebook, PalTalk, AOL, Skype, YouTube and Apple. Congress, following up on its original report, had given the government authority to do this under the Protect America Act in 2007 and the FISA, or Foreign Intelligence Surveillance Act, Amendments Act of 2008, which immunized private companies that cooperated voluntarily with U.S. intelligence collection.

The NSA has also conducted a regular program to monitor phone conversations. The agency and the FBI, now cooperating much as the original Senate-House report urged them to do, won a secret order from the Foreign Intelligence Surveillance Court on April 25, giving the government unlimited authority for three months to amass the telephone records of millions of U.S. customers of Verizon, according a report in The Guardian. Sen. Dianne Feinstein, D-Calif., the chairwoman of the Senate Intelligence Committee, said the order appeared to be merely a "three-month renewal of what has been in place for the past seven years." Feinstein and many other senators defended the program that they themselves set in motion in the last decade. Rep. Mike Rogers, R-Mich., chair of the House intelligence committee, said that "within the last few years, this program was used to stop a terrorist attack in the United States."

Nonetheless, the NSA/FBI programs have raised very real concerns over whether this domestic surveillance has violated constitutional protections of privacy for Americans, despite efforts to restrict the data collection to foreign sources. Intelligence professionals counter that the perception of a Big-Brother-like surveillance state must be balanced against the equally real concerns about tracking terrorists that date back to 9/11, issues that have still not been fully resolved today.

The challenge is that even now, in spite of these programs, the intelligence community remains overwhelmed by data, and as the Boston Marathon bombings in April showed, it is very difficult to piece together clues in time to stop an attack. "There are massive gaps in our ability to actually analyze data. Much of the data just sits there and nobody looks at it," says one former NSA official who would discuss classified programs only on condition of anonymity. "People can do pretty horrific things on their own. Whether with explosive devices, or chemicals or biological agents. Everybody's walking around with these devastating weapons. How are you going to stop that?"

Intelligence professionals say that it is only with mass data collection that they can find the key "intersections" of data that allow them to piece together the right clues. For example, if an individual orders a passport and supplies an address where some suspicious people are known to be, that might raise some concerns – without, however, leading to a definite clue to a plot. Yet if the same person who ordered the passport also buys a lot of fertilizer at another address, then only the intersection of those two data points will make the clues add up to a threat that authorities can act on. In a Jan. 30, 2006 op-ed in The New York Times headlined "Why We Listen," former NSA senior director Philip Bobbitt provided a vivid example of how this "threat matrix" works. On Sept. 10, 2001, he wrote, the NSA intercepted two messages: ''The match begins tomorrow'' and ''Tomorrow is zero hour.'' They were picked up from random monitoring of pay phones in areas of Afghanistan where Al Qaeda was active. No one in the intel community knew what to make of them, and in any case they were not translated or disseminated until Sept. 12. But, Bobbitt wrote, "had we at the time cross-referenced credit card accounts, frequent-flyer programs and a cellphone number shared by those two men, data mining might easily have picked up on the 17 other men linked to them and flying on the same day at the same time on four flights."

In the early years after the 2002 congressional report, the NSA sought to redress this problem, setting up a giant $1 billion-plus program called Trailblazer that was to have brought the agency up to date in such pattern analysis and "data mining." The program was to have transformed the NSA's blizzard of signals intelligence into an easily searchable database.

But Trailblazer turned into such a boondoggle. What went wrong? The NSA, using traditional defense contractors like Science Applications International Corp. (SAIC), sought to do too much at once, applying a clunky top-down solution to what was a Silicon Valley problem, Ed Giorgio, who was the chief codebreaker at NSA for 30 years, told me in an interview in 2006. "The biggest problem with Trailblazer was there was a grand theory of unification that was going to solve the problem, as if the 'central committee' could really do what's best done by a distributed network of people," he said. Fred Cohen, a former computer scientist at Sandia Labs, said then that what the NSA failed at was "to put out enough small money to enough different creative thinkers to explore a lot more possibilities."

Now the NSA has apparently done just that, deploying America's most advanced tech companies in its restless search for threats.

On March 12 of this year, at a hearing of the Senate Intelligence Committee, Wyden asked James Clapper, the director of national intelligence: "Does the NSA collect any type of data at all on millions or hundreds of millions of Americans?" Clapper responded: "No, sir." When Wyden followed up by asking, "It does not?" Clapper said: "Not wittingly. There are cases where they could, inadvertently perhaps, collect—but not wittingly."

In an interview on Thursday, Clapper sought to clarify his remarks. "What I said was, the NSA does not voyeuristically pore through U.S. citizens' e-mails. I stand by that," he told National Journal.

Clapper's earlier denial was broader than that, of course, and his narrower version this week isn't going to satisfy critics -- especially as details of PRISM leak out. But make no mistake: Wyden knew the answers before he asked the question. Wyden, in fact, along with another Democrat on the Senate intelligence committee, Mark Udall of Colorado, has been warning for some time that the programs he helped to promote a decade before had now, in his view, overreached. "I do not take a back seat to any member of this body in terms of protecting the sources and methods of those in the intelligence community," Wyden said in a speech at the end of 2012. But he said those programs "should never be a secret from the American people."

Stay up-to-date with federal news alerts and analysis — Sign up for GovExec's email newsletters.
Close [ x ] More from GovExec

Thank you for subscribing to newsletters from
We think these reports might interest you:

  • Sponsored by G Suite

    Cross-Agency Teamwork, Anytime and Anywhere

    Dan McCrae, director of IT service delivery division, National Oceanic and Atmospheric Administration (NOAA)

  • Data-Centric Security vs. Database-Level Security

    Database-level encryption had its origins in the 1990s and early 2000s in response to very basic risks which largely revolved around the theft of servers, backup tapes and other physical-layer assets. As noted in Verizon’s 2014, Data Breach Investigations Report (DBIR)1, threats today are far more advanced and dangerous.

  • Sponsored by One Identity

    One Nation Under Guard: Securing User Identities Across State and Local Government

    In 2016, the government can expect even more sophisticated threats on the horizon, making it all the more imperative that agencies enforce proper identity and access management (IAM) practices. In order to better measure the current state of IAM at the state and local level, Government Business Council (GBC) conducted an in-depth research study of state and local employees.

  • Sponsored by Aquilent

    The Next Federal Evolution of Cloud

    This GBC report explains the evolution of cloud computing in federal government, and provides an outlook for the future of the cloud in government IT.

  • Sponsored by LTC Partners, administrators of the Federal Long Term Care Insurance Program

    Approaching the Brink of Federal Retirement

    Approximately 10,000 baby boomers are reaching retirement age per day, and a growing number of federal employees are preparing themselves for the next chapter of their lives. Learn how to tackle the challenges that today's workforce faces in laying the groundwork for a smooth and secure retirement.

  • Sponsored by Hewlett Packard Enterprise

    Cyber Defense 101: Arming the Next Generation of Government Employees

    Read this issue brief to learn about the sector's most potent challenges in the new cyber landscape and how government organizations are building a robust, threat-aware infrastructure

  • Sponsored by Aquilent

    GBC Issue Brief: Cultivating Digital Services in the Federal Landscape

    Read this GBC issue brief to learn more about the current state of digital services in the government, and how key players are pushing enhancements towards a user-centric approach.


When you download a report, your information may be shared with the underwriters of that document.