The Obama administration wants students to know how their colleges rate.

The Obama administration wants students to know how their colleges rate. Alexander Image/Shutterstock.com

How a Digital Swat Team Drove Home Obama’s College Scorecard

The administration brought in techies after three years of controversy.

Some problems call for technical wizardry, others demand policy choices that prompt never-ending debate.

This September, the Obama administration released its much-anticipated college scorecard, an ambitious effort to present data to help students pick a school that’s worth their money. For nearly three years, the initiative had bogged down over fears in the higher education community and Congress that it would go too far in “rating” colleges and conditioning financial aid on graduates’ outcomes in earning power.

The Education Department “lacks a clear framework, the data, and the time needed to do this well,” warned Terry Hartle, senior vice president for government and public affairs at the American Council on Education in December 2014. Opposition also came from House Judiciary Committee Chairman Bob Goodlatte, R-Va., and Sen. Mark Warner, D-Va., with Goodlatte declaring, “It is not the place of the federal government, through a ratings system, to attempt to measure the value of an individual’s education.”

The Obama policymakers, led by White House Domestic Policy Council Deputy Director James Kvaal, pressed on, ultimately rejecting plans to rank colleges in the manner of U.S. News & World Report but settling for some groupings of schools that are “low cost that lead to high incomes.”

Their push looped in not just the Education Department but the Council of Economic Advisors, the Treasury Department and the Office of Management and Budget. Obama unveiled the product in a September 2015 radio address. “You’ll be able to see how much each school’s graduates earn, how much debt they graduate with, and what percentage of a school’s students can pay back their loans,” he said, “which will help all of us see which schools do the best job of preparing America for success.”

Less visible to the public and the campus community was the role played by the technical design staff—the White House U.S. Digital Service and the General Services Administration’s 18F swat team.

In an interview with Government Executive, two key players described how they pulled it off, emphasizing their user-centered website design and the modern need to reach a young audience by building first for mobile devices. For them, the policy controversy was a non-issue.

User-Centered Design

“We were brought on knowing what the goal was—getting great metrics for students making informed choices,” said Lisa Gelobter, the Digital Service’s chief digital service officer at the Education Department who started on the scorecard this April. “My mandate was to be able to take data analysis from a software and product development perspective and carve out what made the most sense.”

Aware that the Education Department had already performed deep research, Gelobter and colleagues “went out and talked to students, asking how is it that they go searching and looking for schools.” All in all, the broader scorecard team over the years interviewed some 9,000 students, parents and higher education officials so “we could focus on being user-centered and delivering the right product as efficiently as possible,” she said. One key observation: “The Education Department has great authority and authenticity” as a source of college data, she said. “Students told us they actually trust data when it comes from government.”

Based on the feedback, the software developers then assembled tools and objectives to build two distinct but overlapping products.

One would be a consumer-facing website tool. “The audience was not just students but guidance counselors and parents,” Gelobter said, for whom “we would actually make the data publicly consumable and part of the conversation around higher education.”

The other product was a professionally targeted application programming interface “to make sure third-party developers can actually build tools specific to target populations with our data behind it.” Interested parties could be researchers, journalists and policymakers, she said. Both commercial and nonprofit entities (nearly a dozen are already involved) can click on the data link at the bottom of the public webpage to download the raw data in the universal spreadsheet format called comma separated values.

That format “sets the data free,” making it “palatable to those who can develop products targeting the vulnerable, first generation or low-income students and actually build tools specific to those needs,” Gelobter said.

The database is massive: 7,000 schools, 1900 columns of categorized data points going back 18 years.

The mantra of user-centered design is “build, measure, learn.” That meant the team avoided the old-fashioned “waterfall method that creates a giant list of requirements up front,” said Shawn Allen, an innovation specialist at 18F who joined the scorecard team in July as a front-end design lead. “Before designing the code, we tried out an idea to understand whether it’s working, then readjusted and course-corrected,” he added. Had they started with a list of 100 requirements, “We would have ended up with very different results.”

Like many modern software designers, the college scorecard team focused on mobile first. “In the application and the website, we used paper prototypes, faux scrolling long pieces of paper with the interface sketched out,” Allen said. “If you can make a website work well on a phone first, then you’re applying progressive enhancement to make the site better on screens that are larger,” he said. “The inverse--starting with a desktop or tablet and then shrinking it to work on a phone--is much more difficult.”

Before settling on what the team called a “minimally viable product” for delivery, the software designers showed early versions to students and other stakeholders, Gelobter said. They developed problem statements to encapsulate what the team is aiming for, adjusting the statements based on usability tests on the design with prospective users. They observed “how students actually search for schools, what words appeal, which are scary, which graphics they responded to,” she said. “The entire process was about learning, from a visual design perspective and from a wording and development perspective,” all premised on “continual evolution and improvement.”

Priorities for the finished product were reshuffled at weekly team critique sessions.

Unleashed on the World

As the unveiling approached in September, “we were excited, but it was a pretty hectic week,” said Allen, who continued to tap talents of colleagues on the 18F team. “We spent the last week twiddling little knobs and fixing very small bugs. The focus was mostly on not breaking things that already work,” such as the back-end server infrastructure. “It turned out it could handle the traffic pretty well, and we didn’t hit any snags,” he said.

Out in the contentious higher education community, the reaction was complicated. The presidents of the conservative small private Hillsdale College in Michigan and Grove City College in Pennsylvania complained about not being included because they don’t accept federal aid.

And on the left, Trinity College President Patricia McGuire took to the Huffington Post to complain that in “the tsunami of data about higher education that the Obama administration unleashed on an unsuspecting nation late last Friday, one fact emerges that was already painfully clear: the 'haves' in higher education have quite a lot; the 'have nots’ struggle mightily. The administration presents a data mash-up with limited utility for consumers but large potential for misrepresentation of social realities.”

To Gelobter, the college scorecard’s performance testing and launch “are testament to work of the entire 18F team, which stood behind this project and had been down this route before. That was the confidence-inspiring portion.”

(Image via Alexander Image/Shutterstock.com)

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.