Family Empowerment Media: track record, cost-effectiveness, and main uncertainties

Editorial note and disclaimer

This report assesses the nonprofit Family Empowerment Media (FEM). The project was commissioned and supported by a donor who wishes to remain anonymous. It was conducted in February 2023 over two weeks.

This report attempts to estimate the cost-effectiveness of FEM, mainly relying on two existing cost-effectiveness analyses (from GiveWell and Founders Pledge). We were asked to look into the cost-effectiveness of the organization in an unbiased way, and we started the research without a prior for whether this organization would be cost-effective or not. In this report, we express cost-effectiveness as a function of the cost-effectiveness of cash transfers (like the original models mentioned above). We did not analyze the cost-effectiveness of any other family planning interventions, so we cannot and do not make any claims about how FEM compares to other family planning interventions.

In addition to looking at the cost-effectiveness of FEM’s program, we describe their track record to date, and try to determine the main uncertainties about the program and its (cost-)effectiveness.

We relied on four expert interviews to attempt to answer some of the questions: We interviewed Anna Christina Thorsheim (Executive Director and cofounder of FEM), Andrew Martin (Senior Research Analyst at GiveWell), Rosie Bettle (Applied Researcher at Founders Pledge), and Dr. Mukhtar Muhammad (medically trained expert in social and behavior change communication with a historical focus on family planning programming).

We have tried to flag major sources of uncertainty in the report and are open to revising our views as more information becomes available.

We haven’t attempted to assess the value of different population ethics views, or how those would affect the (cost-)effectiveness of FEM’s work. We believe that that is a highly complex topic that would take more time than the short period we had to conduct this research. Work on this would benefit from the Worldview Investigations Team at Rethink Priorities, which could explore family planning topics in the future.

We would like to mention that Melanie Basnak previously worked at Charity Entrepreneurship (CE), the nonprofit organization that incubated FEM. Melanie did not overlap with Anna Christina Thorsheim during her time at CE and believes she oversaw this research in an unbiased manner.

Executive summary

In a randomized controlled trial in Burkina Faso from 2016-2018, a family planning mass media campaign led by the nonprofit Development Media International (DMI) led to a 5.9 percentage point (20%) increase in the modern contraceptive prevalence rate (mCPR), leading the study authors to estimate a cost effectiveness of $7.70 per couple-year protection, albeit with uncertainty. The nonprofit Family Empowerment Media (FEM) ran a pilot family planning mass media campaign in Kano, Nigeria, and they observed a 6 percentage point (75%) increase in mCPR using less rigorous pre- and post-pilot data comparisons. GiveWell conducted a cost-effectiveness analysis (CEA) of DMI’s intervention, finding it to be 1.1-5.4x the cost-effectiveness of cash transfers, depending on the context. FEM altered the model to pertain to its pilot intervention in Kano, finding that their intervention was ~27x the cost-effectiveness of cash transfers. Founders Pledge conducted a separate CEA, finding a cost-effectiveness of ~22x that of cash transfers, complete with a full (unpublished) report.

We used the original models as a starting point to conduct our own cost-effectiveness estimation, tweaking certain values. In most cases, our adjustments would tend to increase these existing cost-effectiveness estimates, so we are confident that the cost-effectiveness of FEM’s intervention beats that of cash transfers, and we believe that their cost-effectiveness might be higher than GiveWell’s and Founders Pledge’s estimates. However, we would like to see several remaining uncertainties resolved before we would feel confident putting a bound on cost-effectiveness, and we discuss these uncertainties in some depth. For instance, we are highly uncertain regarding the persistence and magnitude of the treatment effect, as well as the nature and magnitude of spillover costs (e.g., to governments) and benefits (e.g., for women’s education, child health, animal welfare, and climate outcomes) at scale. We hope that FEM’s research will take these uncertainties into account, and we believe that further data exploration and conversations with experts could additionally help to reduce them.

Mass media communication to increase contraceptive uptake

A mass media campaign randomized controlled trial run by development researchers led to a 20% increase in the modern contraceptive prevalence rate, at an estimated cost of $7.70 per woman per year

Development Media International (DMI) is a nonprofit organization that uses radio, TV, and mobile video campaigns to elicit behavioral change and improve health outcomes in low- and middle-income countries (LMICs). They run campaigns focusing on a variety of health areas, including child survival, early childhood development, and family planning. Their family planning campaigns provide information about different contraceptive methods and highlight the benefits of family planning. From June 2016 to December 2018, they partnered with development economists to run a randomized controlled trial (RCT) of their mass media campaign in Burkina Faso.¹ They found that their campaign led to a 5.9 percentage point increase (p=0.046) in modern contraceptive² prevalence rate (mCPR) in intervention areas on average, a 20% increase relative to the mCPR of 29.5% in control areas (Glennerster et al., 2022).

The researchers modeled that a scale-up of the radio messaging program at the national level would lead to 225,000 additional women using modern contraceptives, costing $7.70 USD per woman per year, under their favored assumptions. The non-representativeness of their survey sample makes extrapolation difficult, since women surveyed had to satisfy certain criteria (including that they were resident in villages that have no electricity, have fewer than 1500 inhabitants, and are 5 km or less from a health center and 5-50 km from the radio station). They estimate that the survey sample is representative of about 30% of women of reproductive age in their RCT and nationally, using national census data. They therefore make two extrapolation assumptions to identify a lower bound impact estimate (70,000 additional women using modern contraceptives due to the national scale-up, assuming no impact on women for whom the survey is not representative) and what they consider to be a realistic³ estimate (225,000 additional women using modern contraceptives due to the national scale-up, assuming equal impact on women for whom the survey is not representative; Glennerster et al., 2022, pp. 35-37).

The cost-effectiveness estimate of $7.70 per woman per year of contraceptive use is rooted in the equal impact assumption, and additionally assumes that contraceptive use lasts for two years on average. This estimate changes to $11.20 if they include a $3.50 “cost of additional supply” (i.e., of contraceptives and health worker time), which seems low to us in part because Sully et al. estimate that the direct cost per user alone (including contraceptive and personnel costs, but excluding “programs and systems” costs) is $7.75 across African LMICs given that the most prevalent contraceptives are among the most expensive (Sully et al., 2019). The same estimates under the no impact assumption are $24.70 and $28.20 per each year a woman is using contraception, respectively.

Due to lack of time, we focused our efforts on the RCT run by DMI, the sole RCT of which we are aware that tests the role of mass media campaigns in increasing contraceptive uptake. However, mass media campaigns have been used before to increase contraceptive uptake and to elicit other behavioral changes. We refer interested readers to the High Impact Practices in Family Planning (2017) and Founders Pledge (2022b) reports.

Family Empowerment Media uses mass media communication to increase contraceptive uptake in Nigeria

Family Empowerment Media (FEM) is a nonprofit that uses radio communication to enable informed family planning decisions. They were incubated by Charity Entrepreneurship (CE) in 2020. CE recommended mass media campaigns as one of the two most promising interventions to start a family planning organization after spending the 2019-2020 research cycle investigating a variety of interventions, in part motivated by the effects observed in DMI’s RCT (Finetti, 2020). We relied on the information on FEM’s website, and an interview with their Executive Director, Anna Christina Thorsheim, to reconstruct their work to date and their plans moving forward.

FEM works in Nigeria, in partnership with several local partners, one in each of the geographical regions they operate.⁴ They have two different types of radio programs: (1) they broadcast 60-second long advertisements consisting of dramas or personal anecdotes on family planning and contraception ten times a day, every day of the week, (2) they do a weekly regional call-in, drama or story show of duration 20-60 minutes for listeners to ask family planning questions to experts and learn more in depth information (Family Empowerment Media, n.d.). During our conversation with Thorsheim, she mentioned that they have also added a drama that follows three families over eight weeks. They believe that incorrect information about modern contraceptives is an important barrier to adoption, and hope that their radio campaigns can close the information gap and lead to an increased uptake in contraception.

Since their inception, they have:

1. Run a proof of concept. This took place September 2020-February 2021 in the state of Kano, the goal being to assess whether their model was feasible. They conducted a one-week campaign during which they broadcasted a radio ad 140 times, reaching an estimated 2-3 million listeners. They then carried out a small survey which showed that four out of five listeners remembered the ad.
2. Run a pilot. They first researched how best to reach the target audience, trying to identify the main barriers for people in reaching reproductive goals in Kano, and understanding how they consume media. Following the formative research period, they ran a three-month pilot campaign. The Performance Monitoring for Action project (PMA)⁵ found a 75% increase (6 percentage points) in uptake in contraceptives among all women in Kano, and a 90% increase (9 percentage points) among married women in a time period corresponding with their campaign (PMA, 2022). They believe theirs was the only new large-scale intervention in that period.
3. Rolled out the campaign in Kano. They then launched a nine-month rollout campaign in Kano State, ending this month. They reached an ‘audience space’ (people who listened to radio over the last two weeks) of 5.6 million people.
4. Run proof of concepts in three new Nigerian states. The last quarter of 2022, they repeated one-week pilots in three new states, where they expect they reached 10-15 million people in the “audience space.” In early 2023 they have started re-airing the content from these new regions in surrounding states, so far reaching five additional states to those in the proof of concept. They have also started doing in depth formative research to better understand the information needs and audience in new regions, which include interviewing and surveying over 2000 individuals.

1. Work with academic collaborators and conduct a time-series analysis to get a better sense of the changes in contraceptive uptake for the pilot and rollout in Kano.
2. Run a RCT (depending on funding). Last year, FEM built a new technology: a transmitter with a receiving antenna that picks up radio signals, and which turns on and sends different information within a radius of about 4-8 km when their content is on air. This allows them to randomize the areas that receive their intervention. They plan to use this technology to run a full RCT. They are hoping to produce and air content in three new states, most likely in Anambra, Kaduna, and Ondo, for 23 months, reaching an estimated 11 million people. They are partnering with four academics to run this evaluation, and will be assessed by an external survey group.
3. Scale-up. They will re-air the content they have produced during the RCT in nine new states with similar cultural and linguistic characteristics, to reach an estimated 24 million people. Thorsheim mentioned that this should be easy to do logistically and, in expectation, highly cost-effective. They have already conducted a small-scale re-airing project in the south-south as a feasibility test.
4. Run exploration pilots (2024-2025). They want to do a pilot of this intervention in another country, and also a pilot of a "Kangaroo Mother Care + Exclusive Breastfeeding" intervention. If the results from their RCT are positive and show their current intervention is indeed (cost-)effective, they will then scale up to a second country, following a positive pilot. If instead the RCT results prove the intervention was unsuccessful, the exploratory pilots can help FEM determine if they should pivot to another country, another intervention, or stop operations altogether.

FEM appears to be a cost-effective program, though there are several uncertainties about the existing models assessing them

Using GiveWell’s model, FEM estimates that their work in Kano was 26.9x as cost-effective as cash transfers

In May 2022, GiveWell (GW) posted a short report assessing the potential of family planning radio campaigns, primarily basing their analysis on the RCT run by DMI (GiveWell, 2022b). The main component of the report is a cost-effectiveness analysis (CEA) using the information from the RCT as inputs, and estimating the cost per additional couple-year of protection (CYP; GiveWell, 2022a). They estimate that the cost per CYP was ~$175 for the Burkina Faso RCT, could be ~$51 if scaled nationally, and could be ~$61 if scaled to five West African countries (Guinea, Benin, Togo, Sierra Leone and Niger). They also computed the cost-effectiveness in relative terms compared to the cost-effectiveness of cash transfers. They found that the family planning radio campaign could be 1.1-5.4x as cost-effective as cash transfers, depending on the implementation context, which is lower than the bar they use to recommend interventions. They claim to have high uncertainty about their estimate, particularly around the value they give to an additional CYP compared to other global health and development outcomes (see row 17 of GiveWell, 2022a), but they ‘guess that the benefits of a couple-year of protection are substantially smaller than the benefit of a life saved’ (GiveWell, 2022b). Their analysis led them to deprioritize further research into the program.

Using GW’s CEA of DMI with the inputs from their campaign in Kano, FEM found that their work is 26.9x as cost-effective as cash transfers (FEM, 2022). The main difference between the two interventions that explains this difference in cost-effectiveness is the cost per woman reached. For DMI, the cost per woman reached was $5.01 in the RCT, and is estimated to be $0.90-$1.25 at scale. The cost per woman reached for FEM’s intervention (using the data from their Kano pilot) is much lower, at $0.18.

We spent approximately six hours assessing GW’s CEA, and met with GW’s Andrew Martin, who interacted closely with this model. The overall framework of the CEA is logical and includes some important components that other actors might not account for (e.g., government costs and fungibility of funding). However, we have identified several components that could be problematic if attempting to use this model to directly inform the cost-effectiveness of FEM’s program:

• As mentioned above, using the existing models as a starting point and modifying the sections we are most skeptical about, we obtain really good cost-effectiveness metrics with cash transfers as a benchmark. However, both of these groups use a metric that will allow them to compare mass media to other interventions, but have not recently published assessments of other family planning interventions or organizations.¹ We are curious to know how FEM’s intervention compares to others, and think that a good first step is to output from the model the cost effectiveness of family planning oriented metrics (e.g., cost per unintended pregnancy averted).² It is worth noting that Charity Entrepreneurship chose mass media campaigns as one of their recommended interventions after reviewing a list of more than 100 ideas (Finetti, 2020), because they found that they could be comparatively more cost-effective than the other interventions, though they only conducted in-depth reports for the top interventions on their list, and they did not assess programs combining multiple approaches, which are common in the family planning space.

Resolving (some of) our uncertainties leads us to believe that FEM’s cost-effectiveness is likely higher than has been previously estimated (low certainty), though we would need more information to estimate and bound true cost-effectiveness

We have primarily relied on an expert interview in resolving some of our major uncertainties. We spoke with Kano resident Dr. Mukhtar Muhammad, a medically trained expert in social and behavior change communication with a historical focus on family planning programming. Dr. Muhammad was in regular contact with FEM during their initial stakeholder engagement in Kano in 2021, where he was representing the Ministry of Health as a family planning expert, and which FEM attended remotely.

Persistence of the treatment effect seems likely higher than estimates used in GiveWell’s and Founders Pledge’s CEAs, though FEM could do more to understand and facilitate persistence via research and capacity building

We asked Dr. Muhammad about women’s use of contraceptives and whether their ongoing usage depends on contraceptive type. His response led us to believe that women in Kano who adopt contraceptives are likely to use them diligently, primarily for birth spacing. He alluded to a survey that found that women in the region of interest have a preference for implants given their cost-effectiveness over pills or injectables, which require regular (and costly) trips to facilities. He also said that women who use contraceptives generally prefer birth spacing of about two to three years, and will go back on birth control between births until they reach menopause. “You hardly find a woman who started using contraceptives who just stops using it,” including new users, he said citing anecdotal evidence.

Dr. Muhammad appears to be highly knowledgeable about family planning in Nigeria, which leads us to believe even more strongly that GiveWell’s and Founders Pledge’s CEAs likely underestimate benefits, given that they assume only 1.5 and 1.96 years of protection per couple, respectively. In the 2018 Nigeria Demographic Health Survey (DHS), the contraceptive discontinuation rates³ are significantly higher for pills (52%), injectables (55%), and male condoms (35%) than for implants (16%), and the most commonly cited reason for discontinuation is a “desire to become pregnant” (35%), all of which is consistent with Dr. Muhammad’s claims (National Population Commission [Nigeria] & ICF, 2022). However, Figure 7.1 (National Population Commission [Nigeria] & ICF, 2022, p. 130) does not indicate that Nigerian women of reproductive age favor implants over other methods of modern contraception, and Glennerster et al. (2022) do not find a statistically significant effect on implant adoption, which they attribute to their primarily being distributed by NGOs (p. 28).⁴ Of course, the DHS data is now somewhat dated, and we lack RCT evidence on the persistence of treatment effects for mass media programs, so we remain somewhat uncertain despite Dr. Muhammad’s claims (though Dr. Muhammad specified he referred to the preferences in Kano and not all of Nigeria, which could explain the discrepancies). We hope that FEM’s RCT will deeply explore the persistence of their treatment effects and their impacts on family planning outcomes.

On a related note, Dr. Muhammad appeared to critique the lack of a “sustainability component” to the FEM intervention. He mentioned that while he believes that FEM contributed to increasing Kano State’s mCPR significantly in the short term (according to the Multiple Indicator Cluster Survey in 2021; National Bureau of Statistics & United Nations Children’s Fund, 2022), he pointed out that there is currently no investment in states’ capacity to continue the program after FEM leaves⁵, which may be an area for improvement of their program, particularly if the treatment effect is largely driven by increasing the salience of contraceptives and thereby reduces behavior-related non-compliance.

Finally, it seems highly plausible to us that there are not only persistent treatment effects for the women who adopt modern contraceptives during the campaign, but that there may be additional effects via norm shifting and social diffusion. Thus, it is possible we would even see increased treatment effects over time.

While the scale-up states are less populous than Kano, they tend to be more urban, which may increase effect size; however, lower investment in healthcare delivery in scale-up states may attenuate treatment effects in scale-up states relative to Kano

Dr. Muhammad was adamant that results from Kano will likely be significantly different from those in other (even neighboring) states due to population size, a sizable healthcare workforce, and many participating partners. While he says the healthcare system is similar to that of neighboring states, “because of the population [size] you cannot compare it with other states.” Kano State is the most populous state in Nigeria (“List of Nigerian states by population,” 2023), and the population of the city of Kano (4.1 million) is second to only Lagos (Sasu, 2022).

The most populous cities in Anambra (Onitsha, population of 561,000), Kaduna (Kaduna, population of 1.5 million), and Ondo (Akure, population between about 500,000 and 750,000) are significantly smaller than the city of Kano. However, Kano (43.6%) has a lower share of its population in urban areas than Anambra (85.6%) and Ondo (55.8%), and a similar share to Kaduna (41.9%). Percentages are 2021 projections from Global Data Lab (n.d.), which are based on five-yearly DHS data.

This distinction may be important; not only does Glennerster et al. (2022) suggest that impacts in urban areas may be stronger than those in rural areas, but Dr. Muhammad also suggested that women in rural areas may not come into (as much) contact with family planning information nor healthcare personnel who provide sufficient information on the benefits of family planning, particularly given the stronger focus on subjects like immunization. He also alluded to the higher education levels in urban and peri-urban areas than in rural areas, and the fact that more educated women are more likely to be able to seek out information due to access to healthcare personnel as well as, for example, search engines on the internet. He mentioned that most rural Nigerians do not live within 5 km of a functional health facility, citing anecdotal evidence that more than a third of the population of Nigeria are not within such proximity of a facility, let alone one that offers family planning services. For these reasons, Dr. Muhammad would expect the effect size to be higher in urban areas. Hence, we may have reason to believe that intervention effects will be higher in Anambra and Ondo, all else equal.

Dr. Muhammad also mentioned that human resources and the health workforce in Kano are superior to other states⁶, since Kano has “more than 1300 healthcare facilities and a lot of personnel working around the clock.” He mentioned that there has been “an increase in the workforce in Kano State due to the political will of the government and other key stakeholders,” in part due to Kano’s leader’s daughter, who is a doctor and public health specialist. He pointed to the increase in family planning adoption as evidence of this prioritization of family planning under this regime.^7,8

Additionally, with “more than nine [partners] supporting in the area of family planning and reproductive health” and “various donors supporting the state” due to its population size, they are likely better equipped to manage exogenous increases in demand for family planning than other regions. “A lot of states in the North have just 1-2 partners, and some have none,” and there may have been “no time that a jingle was aired to enlighten the community on the benefit of contraception” (i.e., there is likely to be lower awareness at baseline), he said. He thinks Kaduna is the only state with a contraceptive prevalence rate higher than Kano, signaling that Kano is particularly well-resourced to manage family planning needs. While Kano’s mCPR appears to have increased from ~8% to ~14% according to PMA (Figure 2), the baseline mCPR in other states can be as low as 0-2%, he said.

Figure 2: Modern contraceptive prevalence rate in Kano State. From PMA Nigeria (Kano): Results From Phase 3 Cross-Sectional Survey by Performance Monitoring for Action, 2022, p. 1 (https://perma.cc/XZX9-3Q5Z). Copyright 2022 by Performance Monitoring for Action.

Thus, the generalizability of the intervention from Kano to Nigeria more broadly is potentially limited without significant resources devoted to creating more robust healthcare systems in other Nigerian states. If we assume that the 75% intervention effect Thorsheim showed us (based on pre/post comparisons) generalizes to other areas where the mCPR is 2%, cost-effectiveness would dramatically decline since the absolute increase in uptake would fall from ~6% of a much larger population newly adopting to 3.5% of a much smaller population newly adopting. However, FEM is not planning to target states with low mCPR at this stage, and is planning to venture south of Kano. Anambra, Kaduna, and Ondo — the states in which they plan to scale in the coming few years — had a significantly higher mCPR than Kano in 2018 according to Nigeria DHS data (see Figure 7.3 in National Population Commission [Nigeria] & ICF, 2022, p. 131, reproduced below as Figure 3).

Figure 3: Modern contraceptive use by state in Nigeria. From Nigeria Demographic and Health Survey 2018 by National Population Commission (Nigeria) and ICF, 2022, p. 131 (https://perma.cc/9PGX-N8NK). Copyright 2022 by National Population Commission (Nigeria) and ICF.

On the other hand, Dr. Muhammad suggested that the cost of the intervention is likely to be lower in Kano’s neighboring states, citing Abuja, Lagos, and Kano as “high-cost states” (we are unclear on whether this list is exhaustive). He suggested that for the same cost to reach 1 million people in Kano, one could reach 2 million (i.e., double) in neighboring states for the same amount of money. Referring just to radio time, he suggested that while costs may be 100,000 naira (~$217) in Kano for two minutes, the same slot might cost 20,000-30,000 naira in neighboring states.⁹

It is currently unclear to us whether costs in Anambra, Kaduna, and Ondo would also be significantly lower, and how such differences might affect program cost-effectiveness. Thorsheim mentioned that FEM has done some airing in these regions and it was less expensive than airing in Kano, but that they should get a better sense of costs this year, and that final costs will depend on whether they choose to air their messaging during peak hours (which could also impact effectiveness). FEM believes the costs should be pretty comparable considering the lower price they will have to pay for re-airing content they have already used.

Dr. Muhammad also suggested that the potential of the program to coincide with “other campaigns and field activities like door-to-door campaigns, dramas, and attending to women at key life events like the marriage ceremony [...] may increase [uptake] more than in Kano.” Of course, such additional campaigns come with their own costs, so implications for cost-effectiveness depend on the interaction effect of running such simultaneous programming.

As for the program itself, it appears that one approach taken in Kano (to which Thorsheim also alluded) has been to connect with Islamic scholars who share their perspective and capitalize on messenger effects from these religious leaders to remedy misconceptions. Changing the narrative involved informing Islamic scholars of the benefits of family planning in terms of birth spacing. Kano State is primarily Islamic (“Religion in Kano State,” 2022), while the other states that FEM plans to target appear to be either primarily Christian (Zaccheus Onumba Dibiaezue Memorial Libraries, n.d., “Ondo State,” 2023) or more mixed and/or secular (“Religion in Kaduna State,” 2022), limiting the relevance of their pre/post findings to other states as well as the reusability of their content. Dr. Muhammad did mention that religion can act as a major cultural barrier, also citing evangelical Catholicism in North Central Nigeria.

More generally, Dr. Muhammad mentioned that some major barriers to scaling the effects of mass media campaigns are cultural and religious issues, as well as misconceptions. There are “lots of rumors about family planning” such as “taking an implant causes cancer,” and he thinks that FEM’s messaging should try to address these misconceptions in “full collaboration with state experts” to “help women understand the truth.” He also suggested translation not just to English and Hausa, but also to Yoruba and Igbo (local languages).

There appear to be some government efforts to encourage radio broadcasts to include messaging on family planning, so future research should evaluate fungibility of FEM donations and additionality of the program

There appear to be efforts to encourage media organizations “to promote FP as a corporate social responsibility” by “us[ing] their structures and systems to promote FP services through discounts, free airtime, and incorporation of FP into their routine broadcasts” (Federal Ministry of Health, 2020, p. 13). We have not evaluated these or several other coinciding state-level interventions that aim to increase demand, nor other (often seemingly complementary, as suggested by Dr. Muhammad) campaigns (see Federal Ministry of Health, 2020, pp. 13-27, where demand generation interventions can be found on pp. 13-15).

Thorsheim mentioned in conversation that FEM “plans to make all resources available to government stakeholders.”¹⁰

Advocacy for government spending on family planning in Nigeria is another potentially promising family planning intervention

When asked if there are other interventions that are as or more effective than mass media campaigns, Dr. Muhammad said “yes, definitely, it’s not just about [social and behavior change communication] and creating demand.” He referred to a costed implementation plan (CIP) document (Family Planning 2020, n.d.) that is generally divided into “six thematic areas, one of which is demand, [...] that are interwoven — you can’t categorically say one is more important than the other.” He suggested the need for buy-in from the state and other local stakeholders via advocacy for increased budgets and approval of the release of funding for family planning:

If you create demand and don’t address supply, there will be a lower CPR [...] if any intervention will come for demand, I always challenge that intervention to equally look at its design to include the supply arm, not by buying the commodities but by patronizing with those other projects and donors¹¹ to provide supply. By the time you increase demand, if the woman goes to a facility for the method and she can’t get it, she’ll be discouraged, [so] creating demand can backfire. [...] We need equal attention to the advocacy and delivery component.

As lead consultant in Kano State for the review of the CIP, Dr. Muhammad found that 700 million naira (~1.5 million USD) were costed for family planning in the CIP — including partner contributions in cash and in kind — though only 10 million (i.e., 1.4%; ~21,700 USD) were actually reported in the State budget, and mentioned that “Kano has been suffering from contraceptive stock-out for years.” Moreover, he mentioned that Nigeria made a $4 million commitment to family planning in 2012, but “despite approval, release has been an issue” leading “several states to experience stock-out issues.” Hence, advocacy for government spending on family planning may be a critical bottleneck for the success of mass media campaigns in areas with less developed healthcare infrastructure and limited spending on family planning. He mentioned a few specific organizations — Advocacy Accountability Mechanism for Maternal and Child Health in Kano State (AMMKaS), Advocacy Core Group (ACG), and OPTION at the state level, and The Challenge Initiative at the national level — that are working on this issue.

We have not had time to investigate the potential cost-effectiveness of this (extension to the FEM) intervention.

We were able to follow-up with Thorsheim on some of the points raised here by Dr. Muhammad. She mentioned that while in principle FEM could work on addressing supply of contraceptives, doing so would increase the costs of the program, which would likely lead to decreased cost-effectiveness. Therefore, they decided not to add a supply arm to their program, but instead operate in states in which the contraceptive stock-outs are comparatively low. They found that there are ~10-12 states in Nigeria with relatively low stock-out incidence where they would feel comfortable working. Before starting their campaign in Kano, they contacted four supply chain experts from Northern Nigeria to get a better picture of the stock-outs in that region, but given disagreements across experts, ultimately chose to rely on stock-out data showing that Kano only has 5% contraceptive stock-outs (compared to up to 50% in some of the other states). Thorsheim also mentioned that tackling the supply issue involves a different set of capabilities than being able to produce effective social behavior change campaigns and that there are other actors working on this, such that FEM might not be particularly well suited to tackle this component.

What we would do with more time

• Work toward resolving some of our main remaining uncertainties
  • Investigate parallel campaigns in the targeted states to help us understand issues related to fungibility, the ability of the government to supply increased demand, and costs to the government that have not been considered in FEM’s CEA.
  • Ask Thorsheim for a complete breakdown of the budget numbers by state, including those used for Kano; their current budget suggests the costs will be the same across states, and Dr. Muhammad suggested that costs can vary quite dramatically. Compare the RCT budgets with the pilot budget for Kano (and compare the Kano budget to realized costs) to understand the reliability of the budget and whether costs vary dramatically from piloting to scaling.
  • More closely investigate treatment effect persistence, possibly by inferring persistence using data on contraceptive breakdown by state (if such data exists) and/or by indirectly estimating it from the persistence of other family planning information interventions (e.g., community health workers information campaigns).
• Improve our cost-effectiveness assessments:
  • Build our own independent model, incorporating some of the strengths of the existing models, but improving some of the weaknesses (e.g., we would use inputs from FEM’s campaign instead of the RCT, and we would attempt to incorporate indirect benefits).
  • Focus on obtaining cost-effectiveness with respect to family planning metrics, and if possible compare that to existing values for other interventions.
• Interview additional experts
  • If building our own CEA, we would additionally contact experts with more specific expertise, for instance regarding the indirect effects of the intervention.
  • We would also like to interview people involved with other types of family planning interventions in Nigeria to get a better understanding of some of the potential challenges of operating in this space (e.g., get another opinion on whether supply is an issue).

Contributions and acknowledgments

Greer Gosnell and Melanie Basnak were the main authors of this report. Melanie Basnak reviewed and supervised this report. Thanks to James Hu for assisting with the citations. Thanks to Adam Papineau for copyediting and to Rachel Norman for assistance with publishing the report online. Further thanks to Anna Christina Thorsheim (FEM); Andrew Martin (GiveWell); Rosie Bettle (Founders Pledge); and Dr. Mukhtar Muhammad for taking the time to speak with us.

If you are interested in Rethink Priorities' work, please consider subscribing to our newsletter. You can explore our completed public work here.

References

Bearak, J., Popinchalk, A., Ganatra, B., Moller, A.-B., Tunçalp, Ö., Beavin, C., Kwok, L., & Alkema, L. (2020). Unintended pregnancy and abortion by income, region, and the legal status of abortion: Estimates from a comprehensive model for 1990–2019. The Lancet Global Health, 8(9), e1152–e1161. https://doi.org/10.1016/S2214-109X(20)30315-6

Bettle, R., Albinsky, V., & Barnes, T. (2022). Moral weights. Founders Pledge. https://docs.google.com/document/d/1UQIiMCEvsdj17W70e3d40ecvExyU-A9dQq8M2a3CXUw/edit?usp=sharing

Charity Entrepreneurship. (n.d.). Our charities. Retrieved February 2, 2023, from https://perma.cc/2XXD-XURT

Development Media International. (n.d.). What we do. Retrieved February 2, 2023, from https://perma.cc/LRM6-HLXZ

Family Empowerment Media. (n.d.). What we do. Retrieved February 2, 2023, from https://perma.cc/5VCU-HM9N

Family Empowerment Media. (2022). Family planning radio campaigns BOTEC with FEM Kano data. https://docs.google.com/spreadsheets/d/1y_XPPq-XwUVVKnI76SOJPd9-rukL43MYoaCtbvbiMYI/edit?usp=sharing

Family Planning 2020. (n.d.). Costed implementation plan resource kit. Retrieved April 13, 2023, from https://perma.cc/3F6N-MXX2

Federal Ministry of Health. (2020). Nigeria National Family Planning Blueprint, 2020-2024. https://perma.cc/EQ6Q-P83N

Finetti, J. (2020). Family planning – Mass media campaign [Research report]. Charity Entrepreneurship. https://perma.cc/UX7V-D4SS

Founders Pledge. (2022a). FEM CEA. https://docs.google.com/spreadsheets/d/1grladfQvtmcO5ar6GWh6ggmFC3gmmMPlKS1d_oUXvFw/edit?usp=sharing

Founders Pledge. (2022b). Mass media interventions: Shallow investigation. https://docs.google.com/document/d/1PHwGxwBH61kU1tFylkSsUIosLsF3ToHW2-JwEK6-SbQ/edit?usp=sharing

GiveWell. (2017, February). Sayana® Press. https://perma.cc/8HCU-2RGQ

GiveWell. (2022a). Family planning radio campaigns BOTEC. https://docs.google.com/spreadsheets/d/1l2_uAHYD42nRTR-xm3RYd8lLpicGPa4r6MPttHe4Nmc/edit?usp=share_link

GiveWell. (2022b). Family planning radio campaigns. https://docs.google.com/document/d/1WmkkZfcOzjuMocdxlBOPaYizmAU2GpK7wUtHB2wjnG4/edit?usp=sharing

GiveWell. (2023). Cost-effectiveness analysis (version 1). https://docs.google.com/spreadsheets/d/13vDDPzotTI5L3mmZooGz7yYY6n4hRX478QPGETC6KQA/edit?usp=sharing

Glennerster, R., Murray, J., & Pouliquen, V. (2022). The media or the message? Experimental evidence on mass media and contraception in Burkina Faso. https://perma.cc/RU3Q-WYNM

Global Data Lab. (n.d.). % population in urban areas. Retrieved March 2, 2023, from https://perma.cc/Y57N-8FWW

Guttmacher Institute. (2022, March). Unintended pregnancy and abortion worldwide. https://perma.cc/WWL8-DA4Q

High Impact Practices in Family Planning. (2017, October). Mass media: Reaching audiences far and wide with messages to support healthy reproductive behaviors. US Agency for International Development. https://perma.cc/JJ2E-YC27

List of Nigerian states by population. (2023). In Wikipedia. https://perma.cc/JNH6-VJRL

National Bureau of Statistics & United Nations Children’s Fund. (2022). Multiple Indicator Cluster Survey 2021, Survey Findings Report. https://perma.cc/B68A-77WD

National Population Commission (Nigeria) & ICF. (2022). Nigeria demographic and health survey 2018. https://perma.cc/9PGX-N8NK

Ondo State. (2023). In Wikipedia. https://perma.cc/8LE2-E87F

Performance Monitoring for Action. (n.d.). About. Retrieved April 13, 2023, from https://perma.cc/6G5E-PHSN

Performance Monitoring for Action. (2022). PMA Nigeria (Kano): Results from Phase 3 cross-sectional survey. https://perma.cc/XZX9-3Q5Z

Religion in Kaduna State. (2022). In Wikipedia. https://perma.cc/TZZ7-M74X

Religion in Kano State. (2022). In Wikipedia. https://perma.cc/39UT-B7TU

Sasu, D. D. (2022, December). Largest cities in Nigeria in 2022. Statista. https://perma.cc/Y8BU-7GSH

Sully, E. A., Biddlecom, A., Darroch, J. E., Riley, T., Ashford, L. S., Lince-Deroche, N., Firestein, L., & Murro, R. (2020). Adding it up: Investing in sexual and reproductive health 2019. Guttmacher Institute. https://perma.cc/9QTF-SKUP

Zaccheus Onumba Dibiaezue Memorial Libraries. (n.d.). Anambra State. Retrieved March 2, 2023, from https://perma.cc/2WAG-N5BC

Notes

---

1. The two-level RCT first randomized radio distribution among 1400 households (1550 women), 50% of which received a radio. Second, they randomized the introduction of family planning messaging from half of the radio stations in their sample (i.e., eight out of 16 stations), and 50% of the women receiving radios were located within the broadcast range of stations assigned to deliver the media campaign. DMI designed the campaign, which involved 90-second radio messages broadcast 10 times each day (at different times of day, week-to-week), as well as three one-hour phone-in shows per week that were designed to be entertaining. The researchers collected survey data from 7500 women across 461 clinics via two survey waves, and used monthly administrative clinic data on contraception distribution to perform robustness checks given that the survey is not nationally representative. The RCT cost $3.1 million and reached 5.1 million women (Glennerster et al., 2022, pp. 2-3). ↩

2. Contraceptives are usually classified as ‘traditional,’ which includes methods such as withdrawal and periodic abstinence, and ‘modern,’ which includes barrier methods (e.g., condoms), hormonal methods (e.g., injectable), and intrauterine devices.  ↩

3. They argue that the treatment effect is likely to generalize, or even increase, among women for whom the survey is not representative for two reasons. First, most of these women live in urban environments and therefore have more radio access, higher contraception use rates, and more access to information on contraceptives relative to women in rural environments; all of these traits were associated with higher treatment effects in their analysis. Second, they test the robustness of their findings outside of the survey sample using administrative data on distribution of contraceptives in areas that the campaign reached but which are not in the survey sample, finding slightly larger treatment effects in those areas. We are convinced by these arguments, though we also think it is plausible that the treatment effect could be smaller among urban populations due to more baseline access to information and contraceptives, which we think could diminish the margin for impact of the intervention (e.g., since this type of information is already more salient to them without the campaign). ↩

4. According to FEM, their local partnerships are crucial to ensure their content is tailored to the population. ↩

5. PMA is a joint initiative of the Bill and Melinda Gates Foundation, the Johns Hopkins Bloomberg School of Public Health, and Jhpiego to collect relevant public health data, with a strong focus on family planning, through surveys in many countries (PMA, n.d.). ↩

6. Currently, however, those numbers are estimates, and the numbers from the past campaign in Kano are real, so it probably makes more sense to use the existing numbers until the scale-up estimates are better grounded. ↩

7. Founders Pledge’s cost-effectiveness estimate of FEM was 22.2x at the time of our research, but is now ~24x due to some updates to their moral weights.

    ↩

8. Bettle says in the Founders Pledge report (unpublished): “I think it is possible that (if FEM were operating more regularly) people might get ‘put off’ by the adverts- although I also think that this possibility is broadly accounted for in my ‘negative impact’ scenario within the CEA.” We are not sure how much we buy into this reasoning, though it is possible the intervention could backfire if, for example, the FEM intervention creates significant new demand and the healthcare system is not equipped to satisfy the demand, leading women to become discouraged from seeking family planning in the future (as Dr. Muhammad noted), or if the messaging is not sensitive to cultural or religious norms. ↩

9. “Special care” effects refer to the additional attention paid to intervention design and implementation in a RCT, though we feel confident that as much or more attention has been paid to the FEM intervention. General equilibrium effects refer to broader impacts on the economy; while these may exist, we do not think the impacts (e.g., from increased female education and empowerment, improved child health, etc.) will be negative. ↩

10. It is worth noting that we have not assessed the track record or cost-effectiveness of DMI, so it is likely that their content is also highly culturally vetted. ↩

11. We are unsure whether the 20% treatment effect in the DMI study is indeed a good treatment effect estimate (although it is certainly conservative compared to the 75% pre/post difference observed in Kano). The reasons for our uncertainty are twofold.

    First, even if we assume the observed effect is fully generalizable, we are unsure whether the relevant effect size is the absolute effect (i.e., 5.9 percentage points increase) or the effect relative to baseline (i.e., 20% increase). In the Kano pilot, it actually seems that the absolute effect is more relevant, since we see about a 6 percentage point increase (from 8% mCPR before the pilot to 14% mCPR after), whereas a 20% increase on the 8% baseline mCPR would have led to just 1.6 percentage points higher adoption. Using the absolute effect would increase cost-effectiveness, since we would expect about 80% higher contraceptive adoption (i.e., 5.9/3.3 percentage points, unpublished internal CEA from FEM) than in their current model.

    Second, we are unclear on whether a higher baseline mCPR necessarily implies larger treatment effects. To use an extreme example, a 20% increase on a baseline mCPR of 90% leads to a projected post-treatment mCPR of 108%, which is not feasible (the maximum possible percentage effect relative to a 90% baseline mCPR would be 11%). Relatedly, we would expect that higher baseline mCPRs might signal that there are fewer women on the margin, i.e., fewer women who might be willing to adopt modern contraceptives that have not already done so, which would lead to an expected decline in the treatment effect beyond some threshold. Accounting for this possibility would decrease cost-effectiveness, though we are unsure by how much.

    We would guess that the first effect would outweigh the second at the baseline mCPRs under consideration (i.e., 14%-18%) for the three states in which they plan to conduct the RCT. ↩

12. The latter two are the total DALYs averted from the reduction in maternal deaths, fistula, postpartum depression, and postpartum anemia (vi), and the equivalent of that number of DALYs if they were attributable to maternal deaths alone (vii). ↩

13. Child welfare could also be considered a cost, e.g., if the child would have lived a good life, contributed to public goods through taxes, and their birth was not detrimental to parent/sibling welfare. That said, given it is an unintended birth, we would assume the family and children would have been better off had the birth been averted. ↩

14. In 2017, GW published a report on Sayana® Press, a type of injectable contraceptive (GiveWell, 2017). Andrew Martin from GW noted that the report may not be fully up-to-date. ↩

15. The existing models have intermediate steps with some of these metrics (e.g., cost per extra CYP in the GW model), but we think they can be improved. ↩

16. The contraceptive discontinuation rate is defined as the “percentage of contraceptive use episodes discontinued within 12 months” (National Population Commission [Nigeria] & ICF, 2022, p. 133). ↩

17. We would be curious to see data on the contraceptive breakdown by state, perhaps both in terms of the supply available and the preferences of women/couples who adopt contraception. For instance, it may be the case that persistence differs by state if condoms or pills are the primary contraceptive in one state while injectables are the primary contraceptive in another. We have not come across such data in our research thus far. ↩

18. Thorsheim mentioned that FEM plans to conduct work in Kano as long as that is the most cost effective region to improve maternal and child child health and that they expect this to be the case for at least a few more years. ↩

19. Thorsheim believes that this might be the case comparing Kano to other Northern states, but that it is unlikely to be the case comparing Kano to Southern states. ↩

20. He mentioned a longstanding issue with the healthcare system in Kano has been “irregular transfers and postings to other cities,” where women trained in the provision of family planning services may end up in facilities that instead target HIV or tuberculosis. ↩

21. If the changes in the regime that Dr. Muhammad mentions happened at the same time as FEM’s campaign, it is possible that part of the impact we see from the campaign might actually be attributable to this, though we are unsure about the time frame for the actions of Kano’s leader’s daughter, and what those actually entailed. ↩

22. To get a sense of the implications of these cost differentials for program cost-effectiveness, we ran some quick calculations. Dr. Muhammad’s numbers suggest that ten one-minute slots per day for a year would cost about $109/minute 10 one-minute slots per day 365 days in a year ≅ $400,000 per year in radio expenses alone. The total cost of the nine-month Kano pilot was $400,000, which suggests that these numbers are not realistic. With more time, we would like to conduct a CEA complete with cost breakdowns to understand the implications of these sorts of cross-state differences for cost-effectiveness, as well as the opportunities for cost saving measures to improve it. ↩

23. Relatedly, she mentioned that FEM’s “programs aren’t like infrastructure where if the NGO stops providing the service there would be harm done,” and that the “end game for their locations is that they hope the change in knowledge and attitudes becomes common in the communities in a way that they’re now shared through conversations locally.” ↩

24. In other parts of the interview, he also mentioned collaboration with governments (e.g., the Ministry of Health) and pharmaceutical companies to address contraceptive supply and health system strengthening issues. ↩