Agencies Need to Get Savvy About Low-Cost Program Evaluation

It’s more important than ever to get the most impact from every program dollar.

If President Trump’s budget blueprint is any indication, most agencies soon will face significant budget cuts. That makes it more important than ever to get the most impact from every program dollar. To do that, agencies will need to use rigorous program evaluation to learn what works and what doesn’t, and to improve program results.

So how can agencies do the most good with their limited evaluation dollars? They need to get savvy about low-cost evaluation strategies.

Those strategies include, first, making greater use of existing data that agencies and their partners already collect. When these administrative data are accurate and applicable to important questions about program efficacy, using them for evaluation can eliminate the need for more costly data collection approaches that often involve tracking down study sample members and gathering survey data. Greater availability of administrative data for evaluation and other purposes is a theme of the bipartisan Evidence-Based Policymaking Commission, launched by Republican Speaker of the House Paul Ryan and Democratic Senator Patty Murray.

With any form of data use, protecting privacy and confidentiality is essential. Today there are established practices for secure data analysis already in place within the federal government, including restricted-access research data centers operated by the U.S. Census. These approaches could be expanded and address concerns about data privacy—concerns that have contributed to a presumption of “no” when program administrators and researchers seek data access. As economist Maria Cancian has noted, “We need to move to what some have called a “yes, unless” attitude [in the government] where we make data usable and accessible unless there is a strong reason to the contrary.” A “yes, unless” attitude would facilitate low-cost evaluation and could help agencies increase efficiency and reduce the taxpayer burden by identifying which programs achieve the desired impacts and which need reform.

A second strategy for low-cost evaluation is to embed rigorous evaluations into existing programs. This involves little or no additional expenditure and often only modest research costs. Take, for instance, intensive college advising programs designed to help low-income, first-generation students get into and graduate from college, such as the federally-funded TRIO programs. These programs often have more students interested in participating than they have capacity to serve. They also tend to be quite expensive to run, so a key question for programs (and for colleges that use them) is: Are these services worth it? By using a lottery to allocate which students receive the limited number of advising slots, programs can embed a randomized controlled trial into their operations. They can then track student outcomes through and after post-secondary education. This allows agencies and organizations to generate reliable evidence about how well their programs work and whether the results justify the costs.

This strategy is being used by Bottom Line, an intensive college advising program that operates in Massachusetts, New York, and Chicago. It has more applicants than capacity, so one of us (Castleman, with co-researcher Andrew Barr) worked with program leaders to modify its application process to use a lottery to select which students the program would advise. Two years into the evaluation, the research has demonstrated that Bottom Line advising generates a substantial improvement in the rate and quality of students’ college enrollment; early results show that students who were offered Bottom Line advising were 14 percent more likely to enroll at a four-year college or university. By embedding an evaluation into an existing program and using existing data through the National Student Clearinghouse, the researchers were able to demonstrate—inexpensively, in terms of research costs—that Bottom Line is producing effects that appear to justify its substantial program cost. Ongoing research will continue to inform this benefit-cost calculation.

At the federal level, an agency that is leading the way on low-cost evaluation is the U.S. Department of Education. Its Federal Student Aid office tested out alternative means of presenting student loan information to reduce default rates, using existing data to track the results for virtually no cost. And the research arm of the department, the Institute of Education Sciences (IES), is currently conducting a low-cost evaluation using an on-line survey to test alternative ways to present school information to parents to see which is most understandable and useful for making elementary and secondary school choices. IES also launched a competitive grant program to support low-cost, short duration evaluations of education interventions in schools to test different ways to produce meaningful improvements in student education outcomes within a short period, such as a single semester or academic year. The studies are low cost because they require researchers to use existing school records to track student outcomes.

Don’t get us wrong: Traditional evaluation methods will always be valuable and necessary in certain cases. For example, low-cost evaluations that use existing data are constrained by data availability. If those data don’t track outcomes, for instance, it will make low-cost evaluation impossible. Our point is not that one type of evaluation is better than the other, but that both should be in every agency’s toolkit for performance improvement. In fact, low-cost evaluation can complement the traditional approach by allowing many programs, or different versions of a program, to be tested to more quickly identify the effective ones. Those that are effective can then be studied more intensively with traditional evaluation.

So how can federal agency officials learn more about low-cost evaluation strategies? A good place to start is with evaluation staff within their agencies, or at other agencies with experience designing and running low-cost studies. The Census Bureau is another key resource for guidance on data that may be available for low-cost evaluation. Agencies may also want to find research partners at universities who have experience running low-cost evaluations and would be enthusiastic about helping to design and conduct evaluations in partnership with the agency. These researcher-practitioner partnerships can be a valuable way for public officials to leverage external research and data experts.

With spending cuts on the horizon, now is the time for more agencies to get up to speed on low-cost evaluation to help them do more learning and improvement, and do it more quickly. In a time of shrinking budgets, getting the most impact from federal programs—and from research and evaluation dollars—is more important than ever.

Benjamin Castleman is an Assistant Professor of Education and Public Policy at the University of Virginia (UVA) and the Founder and Director of the Nudge4 Solutions Lab at UVA.

Andrew Feldman is a Visiting Fellow at the Brookings Institution and hosts the Gov Innovator Podcast. He previously served at the White House Office of Management and Budget in the Obama Administration.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.