Highlights:
- We review a rare teen pregnancy prevention program—Teen Options to Prevent Pregnancy (TOPP)—that has credible evidence of a major effect on pregnancies and births. A recent federally-funded randomized controlled trial found that TOPP reduced the pregnancy rate by about one-half and the birth rate by about one-third over a 20-30 month period.
- Unlike TOPP, the large majority of teen pregnancy prevention programs that the federal government labels “evidence based” have only been shown to produce effects on so-called surrogate outcomes such as condom use or number of sexual partners. The effects on ultimate outcomes, including pregnancies and births, remain unknown. Because surrogate outcomes are often poor predictors of ultimate outcomes, it is likely that most of these programs would be found not to produce effects on actual pregnancy if further tested in rigorous studies.
- The TOPP findings of large effects on pregnancies and births are therefore exceptional. A replication trial is needed to confirm these results and see if they generalize to other settings.
To establish that a social program or medical treatment produces a meaningful improvement in people’s lives, it is generally not enough to show that rigorous evaluations have found statistically-significant effects. The evaluations must also show effects on outcomes that matter—in other words, that are of policy or practical importance. The history of medicine, for example, is littered with examples of treatments that produced sizable, statistically-significant effects on so-called surrogate outcomes—such as tumor size, smoothed heart rhythms, “good” cholesterol (HDL), or amyloid plaques on the brain—that are intended to predict important health outcomes, and yet failed to produce significant effects on those ultimate outcomes (e.g., survival rates, heart attacks, cognitive ability). A recent analysis found that of the 23 cancer drugs that the Food and Drug Administration approved during 2008-2012 on the basis of a surrogate outcome (such as reduction in tumor size) and which subsequently underwent testing for their effect on overall survival, only five (i.e., 22 percent) had positive effects on survival; the remaining 18 did not. [i]
Similar examples abound in social policy: Education programs rigorously shown to produce sizable, statistically-significant effects on children’s ability to identify letters and words, yet no significant effect of their subsequent reading ability; [ii] programs for young mothers on welfare shown to produce sizable, statistically-significant effects on their receipt of a General Equivalency Degree (GED), yet no significant effect on their employment, earnings, or welfare receipt; [iii] programs to prevent unintended pregnancies shown to produce sizable, statistically-significant effects on women’s intentions to use long-acting contraceptives, yet no significant effect on the rate of unintended pregnancies. [iv]
In the field of teen pregnancy prevention, the large majority of programs that the federal government labels as “evidence based” have unfortunately only been shown to produce effects on surrogate outcomes, such as condom use or number of sexual partners. [v] It is not yet known whether these programs produce effects on ultimate, policy-important outcomes, such as rates of pregnancy, birth, or sexually-transmitted disease—in most cases, because the studies of these programs did not measure those ultimate outcomes. If the history of evaluation is any guide, it is plausible that some of these programs would be found to produce such effects if further tested in rigorous studies, but most would not.
In today’s report, we highlight an exciting break from the usual pattern of claimed effectiveness in teen pregnancy prevention based on outcomes other than teen pregnancy. We showcase the recent publication of a federally-funded, high-quality randomized controlled trial (RCT) of the Teen Options to Prevent Pregnancy program, which found a large effect on actual pregnancy and birth rates. The following is our brief overview of the study, and our full, two-page summary is linked here.
This is a well-conducted RCT of Teen Options to Prevent Pregnancy (TOPP), an 18-month program for low-income adolescent mothers that aims to reduce rapid repeat pregnancy and promote healthy birth spacing. The study, which had a sample of 598 mothers ages 10 to 19, found that TOPP produced sizable, statistically-significant reductions in repeat pregnancy during the 20 months after random assignment. Specifically, 21 percent of the TOPP group experienced a repeat pregnancy versus 39 percent of the control group. In addition, based on state birth records obtained for the subsample of mothers ages 18 to 19 at the start of the study, who comprised 72 percent of the total sample, TOPP had a sizable, statistically-significant effect on their subsequent births during the 30 months after random assignment (24 percent of the TOPP group gave birth versus 36 percent of the control group). These effects may have been driven by the TOPP group’s being significantly more likely to use a long-acting reversible contraceptive (LARC). Forty percent of the TOPP group reported using a LARC in the three months prior to the 20-month survey versus 27 percent of the control group.
Source: Jack Stevens, Robyn Lutz, Ngozi Osuagwu, Dana Rotz, and Brian Goesling, “A randomized trial of motivational interviewing and facilitated contraceptive access to prevent rapid repeat pregnancy among adolescent mothers,” American Journal of Obstetrics and Gynecology, available online June 12, 2017.
As we note in our full summary, the study has one main limitation that would be valuable to address in future research. Specifically, the study was conducted at a single site—a health system serving seven counties in Ohio. Thus as a next step to strengthen the evidence, we would recommend a modest expansion of the program to new sites, coupled with a second RCT to determine whether these new sites can faithfully implement TOPP and replicate the promising effects found in the Ohio site. This step is critical as it is not uncommon that an initial high-quality RCT finds sizable effects, but subsequent RCTs are unable to reproduce these effects elsewhere.
If the new RCT can successfully reproduce the current findings, TOPP would then qualify as one of the few social programs with strong, replicated evidence of sizable effects on important outcomes. Since nearly 20 percent of teen births in the United States are repeat pregnancies, it is reasonable to believe that large-scale implementation of TOPP—in adherence, of course, to the program’s key elements—would have effects on teen pregnancy rates that are of national importance.
This post was co-authored by Howard Rolston, former Director of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services; and Jon Baron, Vice President of Evidence-Based Policy, Laura and John Arnold Foundation.
Response provided by the lead author
The lead study author, Jack Stevens, did not provide a formal written response, but he and his colleagues suggested a few edits to the description of TOPP in our full evidence summary, including aspects of the program needed for successful implementation, that we incorporated into the final version.
References
[i] Chul Kim and Vinay Prasad, “Cancer Drugs Approved on the Basis of a Surrogate End Point and Subsequent Overall Survival: An Analysis of 5 Years of US Food and Drug Administration Approvals,” JAMA Internal Medicine, vol. 175, no. 12, December 2015, pp. 1992-1994.
[ii] U.S. Department of Health and Human Services, Administration for Children and Families, Head Start Impact Study Final Report, January 2010.
[iii] Janet C. Quint, Johannes M. Bos, and Denise F. Polit, New Chance: Final Report on a Comprehensive Program for Young Mothers in Poverty and Their Children, MDRC, 1997.
[iv] Cynthia C. Harper, Corinne H. Rocca, Kirsten M. Thompson, Johanna Morfesis, Suzan Goodman, Philip D. Darney, Carolyn L. Westhoff, and J. Joseph Speidel, “Reductions in Pregnancy Rates in the USA with Long-Acting Reversible Contraception: A Cluster Randomized Trial,” The Lancet, vol. 386, issue 9993, 2015 , pp. 562-568.
[v] Coalition for Evidence-Based Policy, HHS’s Evidence-Based Teen Pregnancy Prevention Program: Excellent First Step, But Only 2 of 28 Approved Models Have Strong Evidence of Effectiveness, 2010 (linked here). U.S. Department of Health and Human Services, Office of Adolescent Health, Evidence-Based Teen Pregnancy Prevention Programs, 2017 (linked here).