Highlights:
- This report highlights new findings from a large, high-quality randomized controlled trial (RCT) of the ASAP program for low-income community college students in Ohio. The study found that the program nearly doubled students’ rate of college graduation three years after study entry.
- The new findings in Ohio reproduce the large impacts on graduation rates found in an earlier high-quality RCT at the City University of New York.
- Significance for higher education: For the first time, policy officials have a proven, replicated tool they can use to greatly increase community college completion by low-income students nationwide.
- Larger policy significance: In a world where few attempts to solve social problems succeed, the ASAP findings show it is possible to identify effective, replicable solutions that actually move the needle.
- More is needed: The number of social programs with such strong, replicated evidence of sizable impacts is small. With a few notable exceptions (such as ASAP), our nation lacks the essential knowledge needed to move the needle on problems such as poverty, educational failure, crime, substance abuse, and homelessness. We offer thoughts on how to build this essential knowledge.
Newly-Reported RCT Findings: ASAP program nearly doubles three-year graduation rates of low-income community college students in Ohio, reproducing large impacts in an earlier RCT in New York. Accelerated Study in Associate Programs (ASAP) is a comprehensive community college program, developed by the City University of New York (CUNY), that provides academic, personal, and financial supports to low-income students. An earlier high-quality RCT of ASAP at CUNY, which we’ve summarized here, found that the program produced a remarkably large, statistically significant impact on college graduation rates—an increase from 41 percent in the control group to 51 percent in the treatment group, as measured six years after study entry.
Now a second (replication) RCT of ASAP, carried out in a different state—Ohio—has also reported large impacts on college graduation rates. The new study, conducted by MDRC with a sample of 1,501 low-income students at three Ohio community colleges, found that the program nearly doubled the rate of college graduation as measured three years after study entry—from 19 percent in the control group to 35 percent in the treatment group (the difference is statistically significant, p<0.01). This three-year effect is almost identical to that found in the CUNY RCT at three years.[i]
Based on careful review, we believe both studies were high-quality RCTs that produced valid results.[ii] In both studies, the effect was driven almost entirely by completion of two-year—rather than four-year—degrees. The program’s cost in Ohio was $8,030 total per student over three years.[iii] (Disclosure: Funding support for the overall Ohio initiative comes from a consortium of philanthropies, but Arnold Ventures funded the three-year follow-up of the Ohio RCT.[iv])
Significance for higher education: This successful RCT replication provides confidence that, if other community colleges faithfully implement ASAP, they too will see substantial graduation gains for low-income students. ASAP is currently the only higher education program with such strong, replicated evidence of sizable impacts on college completion, based on our systematic monitoring of the evaluation literature. Neither ASAP RCT has yet measured the program’s long-term impact on workforce earnings, so it not yet known to what extent the increase in participants’ graduation rates will lead to improved economic outcomes.
Larger policy significance: In a world where few attempts to solve social problems succeed, the ASAP findings show it is possible to identify effective, replicable solutions that actually move the needle. To hear many policy officials and commentators tell it, addressing social problems (such as low college completion rates for low-income students) is just a matter of summoning the political will or allocating the necessary funding. Such a view is at odds with the evidence. As we have discussed in prior Straight Talk reports [a,b], social policy is akin to other fields like medicine and business, where the large majority of treatments and innovations evaluated in high-quality RCTs are found not to produce the hoped-for effects, and there is a need to develop and test numerous approaches to identify the exceptional ones that do work.
The new ASAP findings show that success in building a body of exceptional, proven programs is indeed possible in social policy, just as it is in medicine and business. ASAP is not alone. In earlier Straight Talk reports, for example, we have highlighted other recent successful RCT replications of important impacts, including Per Scholas workforce training and KIPP public charter schools.
More is needed: The number of programs with such strong, replicated evidence of sizable impacts is still small, and the few that exist don’t come close to addressing all policy areas and vulnerable populations. Thus, with a few notable exceptions (such as ASAP), our nation lacks the essential knowledge needed to move the needle on problems such as poverty, educational failure, crime, substance abuse, and homelessness.
We believe the fastest way to build this body of essential knowledge in the near term is to fund and/or conduct replication RCTs of programs with especially promising evidence from prior RCTs. Thus, for example, our team at Arnold Ventures is funding replication RCTs of 12 programs identified on our Social Programs That Work website as having “Top Tier” or “Near Top Tier” evidence of effectiveness.[v]
Ours is a low-hanging fruit strategy: Given the strength of the prior evidence for these programs, we anticipate that most (but not all) of these RCTs will—like the Ohio ASAP study—reproduce their sizable earlier impacts in new settings and study samples, thereby giving policy officials a larger number of proven tools to move the needle on important social problems. As an additional step, we fund many RCTs of programs whose prior evidence is promising but not yet Top Tier or Near Top Tier, and early results suggest that a decent portion of these too—though fewer than half—will show meaningful positive impacts. (We welcome additional RCT grant proposals—see funding announcements a,b,c.)
But in order for our country to make steady, sustained progress on major social problems, a much broader, more systematic effort by government, philanthropies, nonprofits, and others to build, rigorously test, and (if effective) expand many, many, many new program approaches is needed. As we have discussed in a previous Straight Talk report, this would be the 21st century’s answer to President Franklin Roosevelt’s call for “bold, persistent experimentation” in government, and advancing such evidence-based reform is a key focus of our work with policy officials.
The Ohio ASAP findings show that success is possible; proof of concept has been established. What has been done once can be done again, and again.
Response provided by the lead study author
We invited the lead study author, Cynthia Miller, to provide written comments on our report. She said that she appreciated the chance to respond and did not have comments to add.
References:
[i] The impact in the CUNY ASAP RCT decreased between the three- and six-year follow-ups as more control group than treatment group members graduated after the three-year point, but the impact remained sizable (as noted above) and is unlikely to decrease further since few sample members are still in college.
[ii] For example, both studies had large samples, successful random assignment (as evidenced by the highly similar treatment and control groups), minimal sample attrition, and valid analyses.
[iii] This total cost of $8,030 includes the direct cost of ASAP services ($5,521) plus the higher cost associated with ASAP students taking more classes than control group students ($2,510). For reference, the current cost of ASAP at CUNY is $10,200 total per student, which is lower than its initial, start-up cost ($16,000) during the CUNY ASAP RCT.
[iv] Other funders of the Ohio ASAP demonstration include the Ascendium Education Group, the Bill & Melinda Gates Foundation, the ECMC Foundation, the Ford Foundation, the Greater Cincinnati Foundation, Haile U.S. Bank Foundation, KnowledgeWorks, the Kresge Foundation, and Lumina Foundation.
[v] We are funding replication RCTs of the following Top Tier and Near Top Tier programs: Nurse-Family Partnership home visiting, Child First home visiting, Career Academies in high-poverty high schools, Annual Book Fairs in high-poverty elementary schools, Nevada’s Reemployment and Eligibility Assessments for unemployment insurance claimants, Per Scholas workforce training, Transitional Care Model for hospitalized older adults, Teen Options to Prevent Pregnancy, LifeSkills Training substance abuse prevention, PROSPER substance abuse prevention, EAAA Sexual Assault Prevention, and ASAP (two replication RCTs in addition to the Ohio study).