Highlights:
- In recent years, there has been a dramatic increase in the number of high-quality randomized controlled trials (RCTs) of programs aimed at helping low-income young adults attend and complete college.
- A key driver of this growth in postsecondary RCTs has been a major reduction in their cost, made possible by the studies’ use of existing administrative databases (rather than expensive surveys) to measure college outcomes.
- Although high-quality RCTs usually find that the programs being evaluated do not produce the hoped-for effects, the sheer volume of postsecondary RCTs has succeeded in identifying a few programs that do produce sizable effects on college attendance and/or completion by low-income students.
- We highlight five such programs and their key RCT findings: City University of New York’s Accelerated Study in Associate Programs, H&R Block College Financial Aid Application Assistance, Bottom Line, Susan Thompson Buffett Foundation Scholarship Program, and Wisconsin Scholars Grant Program.
- None of these findings are yet definitive, in most cases because longer-term follow-up is needed to see if the effects endure. But researchers seem to be well on their way to building a body of proven programs that can be used to increase college attendance and completion by disadvantaged young adults nationwide.
In recent years, there has been a dramatic increase in the number of rigorous program evaluations in the area of postsecondary education—including large, well-conducted randomized controlled trials (RCTs). A key reason for this increase has been a major reduction in the cost of such studies, made possible by the availability of high-quality administrative data on college outcomes (e.g., enrollment, persistence, completion) of U.S. students.
More specifically—in the old days, an RCT with a sample of 2,000 high school seniors had to measure their college outcomes by attempting to locate and survey each of those students to find out who enrolled in college, who stayed in college, and who completed college. As you can imagine, the cost of such data collection adds up quickly, especially if the students are geographically dispersed and the study seeks to measure not just short-term outcomes such as matriculation, but more important long-term outcomes such as degree completion. Studies conducted in this manner typically cost several million dollars, and therefore, few were carried out.
Today, large RCTs of this type can often be conducted at a fraction of that cost by measuring college outcomes using administrative data from (i) the National Student Clearinghouse, which contains enrollment and degree information for more 98 percent of U.S. undergraduates, or (ii) specific states, colleges, or universities, whose data systems often contain information not only on student enrollment and degree receipt, but also grade point average, credits attempted, and other outcomes.
As an example, at the Laura and John Arnold Foundation, we are funding an RCT of Bottom Line—a program aimed at increasing college attendance and completion by low-income, first-generation students—that has a large sample (2,422 students across three sites in Massachusetts and New York) and long-term follow-up (seven years), yet will cost less than $200,000 from start to finish. This is possible because the key study outcomes—from college matriculation through degree completion— are measured with National Student Clearinghouse data rather than through original (and costly) data collection via surveys. We provide more information about this study and its early findings below.
As we’ve noted elsewhere, most large, high-quality RCTs unfortunately find weak or no positive effects for the programs that are evaluated, and postsecondary RCTs are no exception. But the sheer volume of such studies in recent years has succeeded in identifying a few programs that do produce sizable effects on college outcomes of low-income students. By testing many more programs, the field is, in effect, following the approach urged by IBM pioneer Thomas Watson in a different context more than half a century ago: “If you want to succeed, double your failure rate.”
What follows are highlights of programs for young adults from low- or moderate- income families that have been found in high-quality RCTs to produce sizable effects on college attendance and/or completion, sustained over two or more years. We identified these RCTs through our ongoing monitoring of the evaluation literature, and have carefully reviewed all of these studies to ensure they were well conducted (e.g., with large samples, successful random assignment, and low sample attrition). However, this is not intended to be an exhaustive list of high-quality RCTs showing positive effects on college outcomes.
City University of New York’s Accelerated Study in Associate Programs (ASAP), a comprehensive community college program that provides low-income students who need remedial education with academic, personal, and financial supports, and requires their full-time enrollment. ASAP services cost approximately $14,000 per student. A high-quality RCT with a sample of 896 students found:
- A 10 percentage-point increase in degree completion at study follow-up six years after random assignment (51 percent of the treatment group completed a degree versus 41 percent of the control group, statistically significant p<0.01).
This effect was driven by an increase in two-year associate’s degrees, rather than an increase in four-year bachelor’s degrees. Longer-term follow-up is ongoing.
H&R Block College Financial Aid Application Assistance, a program of streamlined personal assistance in completing a college financial aid application, provided to low- and moderate-income families with a dependent child at or near college age. The cost of the assistance is approximately $100 per program participant. A high-quality, multisite RCT with a sample of 1,045 youth found:
- An 8 percentage-point increase in the rate of college enrollment for two consecutive years, at study follow-up three-and-a-half to four years after random assignment (36 percent of the treatment group enrolled for two consecutive years versus 28 percent of the control group, statistically significant p<0.05). The study and its key findings are summarized here.
Bottom Line, a program that provides one-on-one guidance to help low-income, first-generation students get into and graduate from college. The program cost is approximately $4,000 per student. A high-quality, multisite RCT with a sample of 2,422 students[i] found:
- An 8 percentage-point increase in college enrollment in the second year after random assignment (87 percent of the treatment group were enrolled versus 79 percent of the control group, statistically significant p<0.01).
- A 14 percentage-point increase in enrollment in a four-year college in the second year after random assignment (77.7 percent of the treatment group versus 63.4 percent of the control group, statistically significant p<0.01).
The study is ongoing, and will measure college outcomes, including degree completion, over a seven-year follow-up period.
Susan Thompson Buffett Foundation Scholarship Program, a program for Nebraska high school seniors from low- or moderate-income families that provides sizable scholarships for attendance at any public college or university in Nebraska and requires participation in learning communities that provide academic and other assistance. The program’s cost during the first two years after random assignment is approximately $15,000 per student, although costs continue to accrue for a total of up to five years as students proceed through college. A high-quality RCT with a sample of 2,431 students[ii] found:
- A 7 percentage-point increase in college enrollment in the second year after random assignment (94.8 percent of the treatment group were enrolled versus 87.6 percent of the control group, statistically significant p<0.01).
- A 14 percentage-point increase in enrollment in a four-year college in the second year after random assignment (79.7 percent of the treatment group versus 65.3 percent of the control group, statistically significant p<0.01).
The study is ongoing and will measure longer-term college outcomes.
Wisconsin Scholars Grant Program, which offers sizable annual grants to young adults from low- or moderate-income Wisconsin families who enroll full-time in one of the state’s public universities and have unmet financial needs. The program’s cost during the first two years after random assignment is approximately $5,400 per student, although costs continue to accrue for a total of up to five years as students proceed through college. A high-quality RCT with a sample of 1,500 students found:
- A 5 percentage-point increase in on-time completion of a bachelor’s (four-year) degree, at study follow-up four years after random assignment (21.0 percent of the treatment group completed a bachelor’s degree versus 16.3 percent of the control group, statistically significant p<0.05).
None of the studies described above is yet definitive, in most cases because the study follow-up period is too short to rule out the possibility that the control group members will “catch up” over time—i.e., earn a degree but do so a few years after treatment group members. Indeed, the study with the longest-term follow-up—the ASAP RCT—found evidence of some control group catch-up between the three-year follow-up (when the effect on degree completion was 18 percentage points) and the six-year follow-up (when the effect was a smaller but still impressive 10 percentage points). The ASAP RCT, which did have a long-term follow-up, has the limitation of being a single-site study; thus, a replication RCT in another site would be important to hopefully confirm the initial findings and establish that they generalize to other settings. (Replication RCTs of ASAP are in fact underway—e.g., in Ohio, where the early results are promising.)
Finally, none of the studies has yet measured workforce outcomes, so we cannot yet be certain that the positive effects on college outcomes will lead to longer-term effects on employment, earnings, and job satisfaction.
Despite these caveats, the above findings suggest that education researchers are well on their way to building a body of programs with sizable, sustained effects on college attendance and completion by young adults from disadvantaged backgrounds. In a world where most programs in education and other fields do not produce the hoped-for effects, that would be a valuable contribution indeed.
Disclosure:
The Laura and John Arnold Foundation is the primary funder of the Bottom Line RCT, and we are also funding or co-funding replication RCTs of ASAP in other sites.
References:
[i] The findings described here apply to the first cohort of students—i.e., the 1,429 who entered the study shortly after its inception—for whom two-year outcomes are available.
[ii] The findings described here apply to the first cohort of students—i.e., the 1,003 who entered the study shortly after its inception—for whom two-year outcomes are available.