Promising Practices Promising PracticesPromising Practices
A forum for government's best ideas and most innovative leaders.

Agencies Need to Get Savvy About Low-Cost Program Evaluation

ARCHIVES

If President Trump’s budget blueprint is any indication, most agencies soon will face significant budget cuts. That makes it more important than ever to get the most impact from every program dollar. To do that, agencies will need to use rigorous program evaluation to learn what works and what doesn’t, and to improve program results.

So how can agencies do the most good with their limited evaluation dollars? They need to get savvy about low-cost evaluation strategies.

Those strategies include, first, making greater use of existing data that agencies and their partners already collect. When these administrative data are accurate and applicable to important questions about program efficacy, using them for evaluation can eliminate the need for more costly data collection approaches that often involve tracking down study sample members and gathering survey data. Greater availability of administrative data for evaluation and other purposes is a theme of the bipartisan Evidence-Based Policymaking Commission, launched by Republican Speaker of the House Paul Ryan and Democratic Senator Patty Murray.

With any form of data use, protecting privacy and confidentiality is essential. Today there are established practices for secure data analysis already in place within the federal government, including restricted-access research data centers operated by the U.S. Census. These approaches could be expanded and address concerns about data privacy—concerns that have contributed to a presumption of “no” when program administrators and researchers seek data access. As economist Maria Cancian has noted, “We need to move to what some have called a “yes, unless” attitude [in the government] where we make data usable and accessible unless there is a strong reason to the contrary.” A “yes, unless” attitude would facilitate low-cost evaluation and could help agencies increase efficiency and reduce the taxpayer burden by identifying which programs achieve the desired impacts and which need reform.

A second strategy for low-cost evaluation is to embed rigorous evaluations into existing programs. This involves little or no additional expenditure and often only modest research costs. Take, for instance, intensive college advising programs designed to help low-income, first-generation students get into and graduate from college, such as the federally-funded TRIO programs. These programs often have more students interested in participating than they have capacity to serve. They also tend to be quite expensive to run, so a key question for programs (and for colleges that use them) is: Are these services worth it? By using a lottery to allocate which students receive the limited number of advising slots, programs can embed a randomized controlled trial into their operations. They can then track student outcomes through and after post-secondary education. This allows agencies and organizations to generate reliable evidence about how well their programs work and whether the results justify the costs.

This strategy is being used by Bottom Line, an intensive college advising program that operates in Massachusetts, New York, and Chicago. It has more applicants than capacity, so one of us (Castleman, with co-researcher Andrew Barr) worked with program leaders to modify its application process to use a lottery to select which students the program would advise. Two years into the evaluation, the research has demonstrated that Bottom Line advising generates a substantial improvement in the rate and quality of students’ college enrollment; early results show that students who were offered Bottom Line advising were 14 percent more likely to enroll at a four-year college or university. By embedding an evaluation into an existing program and using existing data through the National Student Clearinghouse, the researchers were able to demonstrate—inexpensively, in terms of research costs—that Bottom Line is producing effects that appear to justify its substantial program cost. Ongoing research will continue to inform this benefit-cost calculation.

At the federal level, an agency that is leading the way on low-cost evaluation is the U.S. Department of Education. Its Federal Student Aid office tested out alternative means of presenting student loan information to reduce default rates, using existing data to track the results for virtually no cost. And the research arm of the department, the Institute of Education Sciences (IES), is currently conducting a low-cost evaluation using an on-line survey to test alternative ways to present school information to parents to see which is most understandable and useful for making elementary and secondary school choices. IES also launched a competitive grant program to support low-cost, short duration evaluations of education interventions in schools to test different ways to produce meaningful improvements in student education outcomes within a short period, such as a single semester or academic year. The studies are low cost because they require researchers to use existing school records to track student outcomes.

Don’t get us wrong: Traditional evaluation methods will always be valuable and necessary in certain cases. For example, low-cost evaluations that use existing data are constrained by data availability. If those data don’t track outcomes, for instance, it will make low-cost evaluation impossible. Our point is not that one type of evaluation is better than the other, but that both should be in every agency’s toolkit for performance improvement. In fact, low-cost evaluation can complement the traditional approach by allowing many programs, or different versions of a program, to be tested to more quickly identify the effective ones. Those that are effective can then be studied more intensively with traditional evaluation.

So how can federal agency officials learn more about low-cost evaluation strategies? A good place to start is with evaluation staff within their agencies, or at other agencies with experience designing and running low-cost studies. The Census Bureau is another key resource for guidance on data that may be available for low-cost evaluation. Agencies may also want to find research partners at universities who have experience running low-cost evaluations and would be enthusiastic about helping to design and conduct evaluations in partnership with the agency. These researcher-practitioner partnerships can be a valuable way for public officials to leverage external research and data experts.

With spending cuts on the horizon, now is the time for more agencies to get up to speed on low-cost evaluation to help them do more learning and improvement, and do it more quickly. In a time of shrinking budgets, getting the most impact from federal programs—and from research and evaluation dollars—is more important than ever.

Benjamin Castleman is an Assistant Professor of Education and Public Policy at the University of Virginia (UVA) and the Founder and Director of the Nudge4 Solutions Lab at UVA.

Andrew Feldman is a Visiting Fellow at the Brookings Institution and hosts the Gov Innovator Podcast. He previously served at the White House Office of Management and Budget in the Obama Administration.

Andrew Feldman is a Visiting Fellow in Economic Studies at the Brookings Institution, a member of the bipartisan Evidence-Based Policymaking Collaborative, and also hosts the Gov Innovator podcast that provides insights for results-focused leaders. He previously served as a Special Advisor on the Evidence Team at the White House Office of Management and Budget in the Obama Administration, including a year-long detail to the U.S. Department of Education. Prior roles include being one of the top three appointed leaders at Wisconsin’s labor department under Governor Jim Doyle; a Senior Policy Advisor in the Office of Governor Doyle; Staff Economist at the White House Council of Economic Advisors in the Clinton Administration; and Special Assistant to the President at MDRC. A native of Milwaukee, Wisconsin, he has an undergraduate degree in economics from Swarthmore College, a masters degree in public policy from the Harvard Kennedy School, and a doctoral degree in public policy from Harvard University.

FROM OUR SPONSORS
JOIN THE DISCUSSION
Close [ x ] More from GovExec
 
 

Thank you for subscribing to newsletters from GovExec.com.
We think these reports might interest you:

  • Forecasting Cloud's Future

    Conversations with Federal, State, and Local Technology Leaders on Cloud-Driven Digital Transformation

    Download
  • The Big Data Campaign Trail

    With everyone so focused on security following recent breaches at federal, state and local government and education institutions, there has been little emphasis on the need for better operations. This report breaks down some of the biggest operational challenges in IT management and provides insight into how agencies and leaders can successfully solve some of the biggest lingering government IT issues.

    Download
  • Communicating Innovation in Federal Government

    Federal Government spending on ‘obsolete technology’ continues to increase. Supporting the twin pillars of improved digital service delivery for citizens on the one hand, and the increasingly optimized and flexible working practices for federal employees on the other, are neither easy nor inexpensive tasks. This whitepaper explores how federal agencies can leverage the value of existing agency technology assets while offering IT leaders the ability to implement the kind of employee productivity, citizen service improvements and security demanded by federal oversight.

    Download
  • IT Transformation Trends: Flash Storage as a Strategic IT Asset

    MIT Technology Review: Flash Storage As a Strategic IT Asset For the first time in decades, IT leaders now consider all-flash storage as a strategic IT asset. IT has become a new operating model that enables self-service with high performance, density and resiliency. It also offers the self-service agility of the public cloud combined with the security, performance, and cost-effectiveness of a private cloud. Download this MIT Technology Review paper to learn more about how all-flash storage is transforming the data center.

    Download
  • Ongoing Efforts in Veterans Health Care Modernization

    This report discusses the current state of veterans health care

    Download

When you download a report, your information may be shared with the underwriters of that document.