Funding What Works: The Importance of Low-cost Randomized Controlled Trials

[Editor's note: The headline and text of this blog have been updated.]

President Obama has emphasized the importance of using rigorous evidence and evaluation to ensure that the government makes smart investments with taxpayer funds. A number of valuable Federal efforts have been launched in recent years under the umbrella of evidence-based policy, such as “tiered evidence” grant programs, Pay-for-Success initiatives, evaluation set-asides, Performance Partnership Pilots, and the establishment of the White House Social and Behavioral Sciences Team. A summary of these efforts is available in the evaluation chapter of the 2014 Economic Report of the President, and a more detailed discussion in the Performance and Management section of the President’s budget. 

Through assessments like low-cost randomized controlled trials (RCTs), the government can analyze data about the success of different programs, thus enabling more informed decisions when allocating funds to maximize the positive impact of these programs in areas such as healthcare, education, and childcare. 

I sat down with Jon Baron, the president of the nonprofit, Coalition for Evidence-Based Policy, to hear about how RTCs work and how RTCs can help the government make better policy.  

Tom Kalil: What is evidence-based policy?

Jon Baron: Evidence-based policy is the general term for efforts to increase the effectiveness of government through the use of rigorous evidence about what works. It includes initiatives to build such evidence by rigorously evaluating the effectiveness of programs and practices, and to use such evidence in decision-making – to focus public funds on those efforts shown to be effective in addressing important problems such as child abuse, low-performing schools, and rising healthcare costs.

Tom: What are RCTs, and why are they important?

Jon: For evidence-based policy initiatives to succeed, policymakers need methods to evaluate the effectiveness of program activities that, first and foremost, are scientifically valid. Well-conducted RCTs are widely regarded as the strongest, most valid method of evaluating effectiveness. Such studies randomly assign a sample of individuals and other entities – such as schools or counties – to a group that participates in a new program (the program group) or to a group that receives services-as-usual (the control group). This process helps ensure that the two groups are equivalent, so that any difference in their outcomes over time – such as student achievement or teen pregnancies – can be attributed to the program, and not to other factors.

Tom: What are you doing to establish the effectiveness of the low-cost RCT?

Jon:  We’ve hosted a competition to analyze the feasibility and value of low-cost RCTs in evaluating what works in public programs. We wanted to demonstrate a new paradigm: that large RCTs, once perceived as inherently too expensive and burdensome for practical use in most areas, can now be conducted in many instances at low cost and minimal burden. Costs can be reduced significantly by measuring key outcomes using data already collected for other purposes – such as student test scores, criminal arrest records, or hospitalization records – rather than engaging in original, and often costly, data collection.

Tom: What did you find as a result of the competition?

Jon: The competition generated a terrific level of interest, attracting over 50 submissions from applicants that included some of the country’s top researchers. The cost of the awarded studies ranges from just $150,000 to $183,000, providing a convincing illustration of both the feasibility and the value of low-cost RCTs.

One of the awarded projects is a large, multi-site RCT of Bottom Line, a program that provides one-on-one guidance to help low-income, first-generation students get into and graduate from college. The study will measure college attendance and completion outcomes using data from the National Student Clearinghouse. The second award will fund a large RCT of Durham Connects, a postnatal nurse home-visiting program designed to improve child and mother health and well-being. The study will use hospital records to measure program impacts on families’ emergency department use and related healthcare costs. The third award will fund a large, multi-site RCT of workplace health and safety inspections conducted by the federal Occupational Safety and Health Administration (OSHA). The study will test whether being randomly chosen for inspection affects businesses’ subsequent injury rates and business outcomes (e.g., sales, business closures) – all measured through data from OSHA and other sources. 

***

The Coalition is one of a number of organizations, like Results for America, the Laura and John Arnold Foundation, and the Annie E. Casey Foundation that are encouraging the Federal government to “fund what works.”  This summer, OSTP and the Coalition will co-sponsor a conference on low-cost RCTs to explore wider government and philanthropic use of such studies. The conference will show philanthropic foundations and state and local governments how they can learn from the Federal government’s use of large RCTs to employ low-cost methods for their own studies.

OSTP wants to hear from you.  What’s your idea for advancing evidence-based government?

Tom Kalil is Deputy Director for Technology and Innovation at the White House Office of Science and Technology Policy