Substandard and falsified antimalarials in sub-Saharan Africa: Health burden and potential interventions

Editorial Note

This report is a “shallow” investigation, as described here, and was commissioned by Open Philanthropy and produced by Rethink Priorities from November 2022 to January 2023. We revised the report for publication. Open Philanthropy does not necessarily endorse our conclusions, nor do the organizations represented by those who were interviewed.

The report focuses on assessing the health burden of substandard and falsified antimalarials in sub-Saharan Africa and discusses current and potentially promising interventions to combat the problem. We initially considered the problem for a variety of drugs across low- and middle-income countries. We then narrowed the scope to antimalarials in sub-Saharan Africa due to better data availability than for other drugs and regions and high rates of substandard and falsified drugs. We reviewed the scientific and gray literature and spoke with six experts.

We do not intend this report to be Rethink Priorities’ final word on substandard and falsified drugs. We tried to flag major sources of uncertainty in the report and are open to revising our views based on new information or further research.

Executive summary

The evidence base on substandard and falsified (S&F) drugs is poor. The best available data has focused on antimalarials in sub-Saharan Africa (SSA). Therefore, this report focuses on S&F antimalarials in SSA, which also appear to have a particularly high prevalence (~19%). S&F antimalarials negatively affect health mainly through reduced treatment effectiveness. We estimate that they cause about 40k-160k deaths and 1M-3M DALYs per year in SSA (80% confidence interval), corresponding to up to one out of four deaths and one out of five DALYs due to malaria in SSA. If we also consider S&F antimalarials’ potential to contribute to widespread artemisinin resistance in SSA, we estimate the health burden could increase by up to 50k deaths and 600k DALYs.

Our best (low confidence) guess is that among S&F antimalarials, 10% are falsified (in the sense that no active pharmaceutical ingredient [API] is present), 5% are substandard but essentially ineffective (due to an API too low to have any effect), and 85% are substandard but still have some, albeit reduced, effectiveness. The high percentage of S&F antimalarials implies that the most promising interventions are likely related to improving quality assurance in manufacturing and storage/transportation rather than to mitigating criminal intent.

Several features of the market for antimalarials in SSA make us pessimistic about tractability. We think it is likely that most S&F ACTs are sold in the private sector (low-confidence estimate: 65%-85%), with 15-20% (70% confidence interval) higher prevalence than in the public sector. Private pharmacies and informal markets will likely be hard to affect in SSA without strong state capacity and political will. The market is also highly fractured, with ~15 prequalified manufacturers and 140-210 (75% confidence interval) manufacturers of non-prequalified ACTs globally.

We created a table that shows our assessment of 24 potential interventions, including our thoughts on tractability and some information about who is currently working on this research and whether there is room for more funding. Our sense is that the most prominent players are the Bill and Melinda Gates Foundation (BMGF), WHO, and United States Pharmacopeia (USP), which is associated with USAID. A low-confidence estimate of total philanthropic spending is $30m-$50m per year, with a large proportion of the money being spent by BMGF on strengthening (regional) regulatory systems.

Overall, we did not find any single intervention we thought would be highly effective and tractable. Two of our top suggestions would be to:

• Limit purchasing of S&F ACTs by consumers through funding subsidized high-quality ACTs in the public sector in order to crowd out S&F ACTs and reduce the overall cost of drugs. It’s possible that this could be achieved by supporting Living Goods’ program, and an RCT was conducted between 2016 and 2021 to measure its impact (results unpublished at the time of writing).
• Collect data to inform interventions, by extending a BMGF-funded prevalence study to more countries or augmenting the types of data collected.

Limited data points to S&F antihypertensives potentially having a similar burden to S&F antimalarials. As most antihypertensives are used in LMICs outside of SSA, the tractability of potential interventions might be higher in those countries due to a possibly higher state capacity to support interventions. Future work could explore this issue in more detail.

Key uncertainties

• Is the burden of S&F antimalarials potentially higher than we expect? It might be that (1) we underestimated the burden and/or likelihood of artemisinin resistance, (2) the burden of S&F drugs will increase further over time (e.g., due to rising online sales of drugs that are largely unregulated), (3) the burden of other S&F medicines (e.g., antihypertensives) is potentially higher than the burden of S&F antimalarials or (4) the underlying data from the reviews we rely on is incomplete and fails to capture hotspots where prevalence of S&F antimalarials is much higher.
• Do donors like Global Fund and USAID have more ability to generate political will, strengthen institutions, and/or affect the private sector than we recognize in our report? Our understanding is limited, and it is possible that working with major donors could be a way to make interventions more tractable.
• Are there specific countries in SSA where political will is high, but funding is the major bottleneck? We may have missed these in a shallow investigation. A conversation with someone at the WHO working on regulatory systems strengthening might provide some ideas.

Scope of this report

Definitions and terminology

We use WHO definitions for poor-quality medicines (WHO, 2017, p. 1):

• Substandard medical products: “Authorized medical products that fail to meet either their quality standards or their specifications, or both.”
• Falsified medical products: “Medical products that deliberately/fraudulently misrepresent their identity, composition or source.”¹

Hamilton et al., 2016WHO, 2017

Theory of change: substandard & falsified drugs increase morbidity and mortality

The theory of change for this report rests on the idea that substandard and falsified medicines can have negative health impacts, such as increased “risks of morbidity and mortality by prolonging illnesses and heighten[ing] the risk of treatment failure, poisoning, and adverse drug interactions,” as well as an increased risk of drug resistance and economic losses (Ozawa et al., 2018, p. 2) (see also Figure 1 below). This problem is likely most common in low- and middle-income countries where populations are especially vulnerable due to a high burden of disease and limited economic resources. In this report, we focus on the health impacts.

Figure 1: Impact of substandard and falsified medical products

Note. From WHO (2017, p. 15).

Prevalence and data availability are highest for antimalarials, and in sub-Saharan Africa

[Confidence: Medium. We are confident that we found the best available prevalence data. However, due to limited data availability, it is possible that some regions or medicines with an even higher (or lower) prevalence of S&F drugs and resulting health burden might be excluded/underrepresented in the literature. Talking to three experts has not given us much more insight.]

Due to a high prevalence of substandard and falsified (S&F) drugs and comparatively high data availability, we focus on antimalarials in sub-Saharan Africa (SSA) for the remainder of this report:

• Data availability on S&F drugs is poor. Most studies are centered on only a few countries and drug types, have small sample sizes, and are based mostly on convenience sampling rather than random sampling (Ozawa et al., 2018).
• Several recent systematic reviews find that the prevalence of S&F drugs² is highest for antimalarials (11.8%-19.7%)³ and in SSA (18.7%-18.9%)⁴ (Ozawa et al., 2018; Ozawa et al., 2022; WHO, 2017). Given these data points, we expect that the prevalence of S&F antimalarials in SSA is ~19% or possibly more.⁵
• See Figure 2 below for examples of findings from WHO (2017).
• Most existing studies are focused on antimalarials (44% of studies) and SSA (50% of studies) (Ozawa et al., 2018). S&F prevalence studies tend to be clustered in certain geographical regions and medicines for several reasons (WHO, 2017, p. 9; conversation with Harparkash Kaur, Assistant Professor at the London School of Hygiene & Tropical Medicine):
  • Priors where researchers expect the prevalence of poor-quality medicines to be high⁶
  • Public health importance of certain medicines/disease burden
  • The convenience of carrying out a study, e.g., due to pre-existing networks
• Very limited data suggests that S&F antihypertensives might have a burden in a similar ballpark compared to S&F antimalarials:
  • Ozawa et al. (2022) find a prevalence of S&F antihypertensives of 20.5%, but this is based on much less data compared to S&F antimalarials.^7,8
  • If we assume 1 billion hypertensive people in LMICs (Sabapathy et al., 2023, p. 1), roughly 30% of whom take antihypertensives (NCD-RisC, 2021, p. 961) and an average S&F prevalence of ~5-10%⁹, then ~15M-30M individuals are exposed to S&F antihypertensives in LMICs. The number of exposed individuals is in a similar ballpark to those exposed to S&F antimalarials.¹⁰ Given a similar burden to S&F antimalarials in terms of the number of people affected, our guess is that the health burden of S&F antihypertensives might be somewhat lower as we expect that malaria is more deadly per case and kills people at a much younger age than hypertension, leading to more years of life lost. However, we have not seen any estimates on this issue and have not tried to estimate it ourselves.
  • Andrew Moran (Director of Global Hypertension Control at Resolve to Save Lives [RTSL]) expressed concerns about S&F antihypertensives in an email exchange and explained that RTSL is currently supporting some work on S&F antihypertensives in Nigeria, particularly quality audits of pharmaceutical manufacturers done by an NGO Quamed.
• Overall, we think that S&F antimalarials likely have the highest prevalence and resulting health burden relative to other regions and medicines. Given limited data, it is possible that underrepresented regions and medicines might have an even higher burden. Further work could explore the importance and tractability of S&F antihypertensives.

Figure 2: Observed failure rates by therapeutic category

Note. From WHO (2017, p. 7).

Artemisinin-based combination therapies (ACTs) are the first-line treatment against malaria and a prime suspect for falsification

Artemisinin-based combination therapies (ACTs) are currently recommended by WHO as the first-line treatment for Plasmodium falciparum malaria.¹¹ ACTs are a combination of an artemisinin derivative (e.g., artesunate) and a partner drug.¹² Both compounds in ACTs function to eliminate the Plasmodium parasites from the bloodstream (WHO website, 2022).

ACT drugs are viewed as a “prime suspect for counterfeiting” since artemisinin is significantly more expensive to produce compared to older, synthetic forms of malaria medicine (Nyqvist, 2020, p. 7; Kaur et al., 2016, p. 1).

Burden of substandard and falsified antimalarials in sub-Saharan Africa

Limited data suggests that prevalence of S&F antimalarials might be highest in West and Central Africa

[Confidence: Low. The vast majority of prevalence data on S&F drugs in SSA comes from only three countries (Nigeria, Tanzania and Ghana), and prevalence studies are likely biased and not representative, making a cross-country comparison difficult.]

Based on a conversation with Michael Deats (Honorary Visiting Research Fellow in tropical medicine at Oxford University) and limited prevalence estimates, we think that the prevalence might be highest in West and Central Africa, but we are highly uncertain about this:

• The best overview of the prevalence of S&F drugs across countries in SSA we found is shown in Figure 3 below.^13,14 This depiction might be a good proxy for the prevalence of antimalarials, as ~40% of the available prevalence data is based on antimalarials (Ozawa et al., 2018).
• The available data is not well-suited for a cross-country comparison because:
  • Within SSA, the vast majority of data on S&F drugs is from only three countries (Nigeria, Tanzania, and Ghana) (Kaur et al., 2016, p. 2; Ozawa et al., 2018).
  • Most prevalence studies are likely not representative, but S&F prevalence studies tend to be concentrated in countries where researchers have prior beliefs that prevalence is high.

Figure 3: Reported national prevalences of substandard and falsified medicines

Note. From Ozawa (2018, p. 11).

Substandard medicines are roughly 6-10 times more prevalent than falsified medicines

[Confidence: Low to medium. Limited available literature finds unanimously that substandard medicines are more prevalent than falsified medicines. However, this finding might be partially because falsified medicines are much more difficult to detect as it is hard to prove deliberate fraud.]

We found little evidence on the relative prevalence of substandard vs. falsified drugs (Hauk et al., 2021) as many studies report the prevalence of substandard and falsified drugs jointly, and there is no clear consensus on criteria for identifying falsified medicines¹⁵ and substandard medicines.¹⁶

The available evidence suggests that substandard medicines are 6-10x more prevalent than falsified medicines, but this might be an overestimate as falsified medicines are difficult to detect:¹⁷

• According to a systematic review by Ozawa et al. (2022), 13.8% of failed drug samples were likely falsified based on active pharmaceutical ingredient (API) levels < 50%. 12.5% of tested samples had an API level of 0%. The authors found that 86.2% of failed samples were likely substandard, i.e., they either contained 50 % ≤ API < 80% or “failed other quality tests (e.g., disintegration, dissolution, degradation, presence of impurities, visual and physical inspection), contained API levels > 80% but below pharmacopeia standards or had API levels > 100%.” These results are based on all S&F drugs (i.e., not only antimalarials).
• Hauk et al. (2021) found 16.5% of tested medicines to be substandard and 1.7% to be falsified (i.e., have evidence of fraud) based on 601 tested samples in Cameroon, DRC, and Malawi.
• Individual studies suggest that substandard antimalarials are also more common than falsified antimalarials, though it is unclear by how much, e.g.:
  • Yeung et al. (2015) found that out of 291 samples tested in Cambodia, >30% were substandard, and 0% were found to be falsified.
  • ACT Consortium Drug Quality Project Team (2015) found that out of 1,717 samples tested in Tanzania, none were falsified, and few were of poor quality.

Our best (low confidence) guess is the following distribution among S&F antimalarials (see Table 1):

Table 1: Estimated relative distribution of substandard and falsified antimalarials

Category	% of S&F antimalarials (mainly based on Ozawa et al., 2022)	% of API (mainly based on Ozawa et al., 2022)	Avrg. treatment effectiveness (based on findings described here)
Falsified (defined as API = 0%)	10%	0%	0%
Substandard - ineffective (defined as API > 0%, but too low to have any treatment effect)	5%	1-20% API	0%
Substandard - partially effective (defined as API > 0% and some treatment effect)	85%	20-95% API (~avrg. API of 40-70%)	60-90%

Note. The estimates in this table are low confidence and based on a combination of our own judgment calls with the available data. We chose the definitions in this table for simplicity reasons; they do not directly correspond to the technical definitions of substandard and falsified drugs, e.g., a drug can be falsified in ways other than API = 0%, and a substandard drug can also have API > 100%. We have not considered those cases here due to a lack of data.

These findings imply that the most promising interventions are likely related to tackling issues with inadequate quality assurance in manufacturing and storage/transportation rather than with criminal intent.

S&F antimalarials can negatively impact health by reducing the effectiveness of drugs and increasing antimicrobial resistance and impurities

S&F antimalarials can have reduced effectiveness

[Confidence: Medium. We are confident that a reduced effectiveness of drugs is common and harmful. There seems to be relatively little data on the relationship between a reduced API and treatment effectiveness, but we are fairly confident that a reduction in API translates into a less than proportional reduction in treatment effectiveness.]

We think that the API (active pharmaceutical ingredient) content is the most important quality attribute of S&F medicines because:

• The API of a medicine is “highly associated with its therapeutic efficacy” (Ozawa et al., 2022).
• A systematic review (McManus & Naughton, 2020, p. 4) found that an “inadequate amount of active pharmaceutical ingredient is the most common stated problem” in the studies reviewed on S&F (see Figure 4 below).

Our best guess is that a 50% reduction in API concentration leads to a 20-30% reduction in treatment effectiveness:

• The relationship between API concentrations and treatment efficacy is non-linear. Typical ACT regimens contain between 2.4-4 mg/kg artesunate (Nosten and White, 2007, p. 185; WHO, 2015, p. 222). Two mg/kg of artesunate are needed for the maximum effect (on average),¹⁸ and roughly 1-1.6 mg/kg of artesunate gives 50% of the maximal effect.¹⁹
• Our (low confidence) guess is that less than about 0.5 mg/kg of artesunate, which corresponds to 13-20% of API, would render the treatment essentially ineffective. We have not seen any concrete data to back up this guess.
• Most studies estimating the health burden of S&F drugs typically make assumptions about the relationship between API concentration and treatment effectiveness (i.e., not based on actual data) (see Appendix A for an example of typical efficacy assumptions in the literature). However, according to Nicholas White (Professor of Tropical Medicine at the University of Oxford), those assumptions are likely overly pessimistic.

Other quality attributes of medicines (e.g., dissolution, degradation) also affect treatment efficacy, but these attributes are little researched, and it is unclear how common they are:

• Dissolution failures can reduce the effectiveness of an antimalarial.²⁰ According to a systematic review by McManus & Naughton (2020, p. 4), dissolution failure might be less common than a reduced API concentration (see Figure 4 below). However, this finding might simply be because dissolution studies are difficult to implement and rarely done.²¹
• Degradation can also reduce the effectiveness of antimalarials, but it does not render them toxic.²² Degradation studies are very limited. We do not have a good sense of how common the issue is, but some evidence suggests that some antimalarials classified as having a reduced API concentration are, in fact, degraded.²³ This evidence suggests that substandard antimalarials are at least partly due to storage issues rather than manufacturing issues.²⁴ Medicines can only be classified as degraded when degradation products are detected in the formulation.

Figure 4: Frequency of six different reported issues concerning the quality of tested medicines

Note. From McManus & Naughton ( 2020 ).

S&F antimalarials can potentially contribute to a widespread artemisinin resistance in SSA

[Confidence: Low to medium. We are confident that widespread artemisinin resistance in SSA would likely have large death and DALY burdens. However, we have a high uncertainty around how likely this scenario is, how quickly it could occur, and the extent to which S&F antimalarials would drive this scenario, as very few estimates exist in the literature.]

Widespread²⁵ artemisinin resistance in SSA — if it were to occur — could cause an estimated 40k-400k deaths and 1M-5M DALYs annually:

• Widespread artemisinin resistance likely has large negative health impacts, as “there is currently no good alternative to ACT suitable for large-scale implementation” (Lubell et al., 2014, p. 2), and there is already resistance to most other antimalarials (CDC, 2018).
• We found two estimates of the death and/or DALY burden in the case of widespread artemisinin resistance in SSA in the literature and received one estimate from a major expert in the field (see findings in Appendix D):²⁶
  • Estimates from the literature suggest ~40k to ~400k deaths/year and ~2.6M DALYs/year. As we found only one DALY burden estimate in the literature and we have some uncertainty around this estimate, we think it is reasonable to assume 1M-5M DALYs.
  • Nicholas White estimates ~400k deaths/year.

Our best guess is that 20 years from now, S&F antimalarials will cause 2-12% of yearly deaths and DALYs from widespread artemisinin resistance in SSA. Using current estimates of the potential burden of resistance, this would mean ~1k-50k deaths and ~20k-600k DALYs per year:²⁷

• We have not found any estimates (neither from experts nor from the literature) of how likely the scenario of widespread artemisinin resistance is, how quickly it can occur, and to what extent S&F antimalarials could be a major driver.
• We guess that the chance of widespread artemisinin resistance to occur in SSA within the next 20 years is 20-40% (70% CI) (30% point estimate):
  • Reasons to believe this will happen:
    • There is already “resistance to nearly all of the [...] currently available antimalarial drugs” (CDC, 2018). Resistance to chloroquine and amodiaquine developed quite quickly — likely within a few decades (e.g., Sá et al., 2012).
    • There are confirmed cases of ‘partial’ artemisinin resistance in both Southeast Asia and SSA (Eritrea, Rwanda, Uganda) (WHO, 2022). Cases of partial resistance emerged in South East Asia within only 5-15 years of ACT introduction, depending on the country (Scott et al., 2018, p. 4).
    • According to some researchers, we might be “at the verge of clinically meaningful artemisinin-resistance” in SSA (e.g., Rosenthal, 2021)
    • Resistance can develop in two different ways: It can either develop independently in SSA or spread from Southeast Asia to SSA due to migration.
• Reasons to believe this will not happen:
  • Moore et al. (2022) found evidence that artemisinins are unlikely to be affected by resistance; they made this discovery by investigating the mechanisms through which artemisinins kill malarial parasites.²⁸ However, there can still be resistance in partner drugs of artemisinin.
  • There are no reported cases of ‘full’ artemisinin resistance to date, and ‘partial’ resistance is unlikely to affect morbidity and/or mortality (WHO, 2022).²⁹
  • It is unknown whether ‘partial’ artemisinin resistance could further evolve into ‘full’ resistance (WHO, 2022).

Many different factors drive the spread of resistance. Our best guess is that S&F antimalarials contribute 10-30% (70% CI) to widespread resistance in SSA:

• Several factors likely play a role in the emergence and spread of resistance. For a full overview of the drivers of resistance, see Appendix E. Some key drivers (which are present in SSA) are, in no particular order (WHO, 2022):
  • Poor treatment practices (e.g., under-dosing, treatment of unconfirmed cases, use of non-pharmaceutical forms of Artemisia)
  • Inadequate treatment adherence
  • Misuse and overuse of artemisinin-based monotherapies
  • Substandard and falsified drugs
• Little is known about the relative importance of drivers of resistance:
  • According to WHO (2022b, p. 12), “[t]here is good consensus on the factors that may drive the emergence and spread of resistance, but knowledge of their relative importance is limited by the lack of available data and evidence.”
  • In Southeast Asia, factors other than S&F ACTs may be the most important drivers of resistance,³⁰ but it is unclear whether this can be extrapolated to the SSA context.³¹
  • According to Nicholas White, S&F antimalarials are a “potentially important one” among different factors driving resistance.

S&F antimalarials can potentially be toxic due to impurities, but this is likely rare

[Confidence: Medium. So far, all evidence we have come across points to the problem of toxic impurities being relatively small. However, we have found little research addressing this topic, and we do not expect that 10 more hours of research would substantially change our views on this issue.]

According to a systematic review by McManus & Naughton (2020), impurities are rarely found in S&F drugs (see Figure 4 above), but this might partly be because they are little studied. According to Michael Deats, toxic impurities are likely uncommon because those who falsify drugs do not have an incentive to create adverse reactions in drugs, as it would draw unwanted attention to the falsification. Deats also pointed out cases where paracetamol or antibiotics were added to falsified antimalarials and vaccines. Adding paracetamol or antibiotics provided a therapeutic reaction to falsified drugs, which made consumers believe that the drug was real (i.e., that it had the stated therapeutic effect).

Onuh et al. (2022) surveyed 365 consumers in several countries in SSA on S&F drugs and found that 81% of those who consumed S&F drugs did not experience “any side effect as an aftermath of drug consumption.” For individuals who experienced side effects, “some of these medical problems were due to drug-drug interaction, wrong medications, self-medications, etc.”

Our best guess is that S&F antimalarials are responsible for roughly 1M-3M DALYs and 40k-160k deaths annually in sub-Saharan Africa. These figures could increase by up to 600k DALYS and 50k deaths if we consider a potential widespread artemisinin resistance.

[Confidence: Medium. Limited estimates exist for DALYs and deaths and many parameter inputs are highly uncertain. However, we think it is unlikely that the estimates are off by an order of magnitude.]

Our current best guess is that S&F antimalarials in SSA cause ~1M-3M DALYs and ~40k-160k deaths annually (if we ignore the potential of widespread drug resistance):

• We summarized all six high-quality modeling studies we could find that estimate the death and/or DALY burden of S&F antimalarials in SSA (see Appendix C). Two of those studies focus on SSA as a whole, and the remaining four studies focus on specific countries in SSA (Uganda, Nigeria, Zambia, and DRC). Only two studies estimate DALYs.
  • Their findings (extrapolating some of their findings using our own calculations) suggest that S&F antimalarials cause roughly 1M-3M DALYs and 40k-160k deaths annually in SSA (80% CI).
  • As far as we can judge, these models are excellent in terms of their setup, but there is a high amount of uncertainty regarding many inputs. We have not been able to judge precisely how sensitive the findings are to some inputs, as the models are very complex, and we have not had direct access to any of them. Our impression is that the estimates are highly sensitive to the assumed case fatality rates, as these vary substantially across estimates.³²
  • We found only one study that estimated the DALY burden of S&F antimalarials across SSA (WHO, 2017) and one other study that estimated the DALY burden in Uganda. These estimates depend on assumptions on how to translate deaths into DALYs, which is done differently across organizations. For example, Open Philanthropy and GiveWell’s methodology assumes a higher number of years of life lost.³³ Thus, using these assumptions, one might estimate a higher DALY burden of S&F antimalarials compared to the estimates made by WHO (2017, p. 53).³⁴

Combining the estimated health burden with a potential widespread artemisinin resistance could increase the health burden by ~1k-50k deaths and 20k-600k DALYs (as explained here).

For context — Malaria prevalence and health burden in sub-Saharan Africa:

• 234 million malaria cases in the WHO African region in 2021 (WHO, 2022, p. 18)
• In 2019, malaria caused almost 600,000 deaths and about 43 million DALYs lost in sub-Saharan Africa (GBD Results Tool, 2019)
• 76% of malaria deaths in children aged < 5 globally in 2021 (WHO, 2022, p. xxi)

Potential interventions

Some features of the market for ACTs make us pessimistic about tractability

Top-line findings:

• Most S&F ACTs (low-confidence estimate: 65%-85%) are probably being sold in the private sector, and this sector is likely to be hard to affect in SSA.
• We are uncertain about procurement processes for the public sector, but from first principles, it seems likely that tractability will remain somewhat limited due to reliance on the same national institutions.
• The ACT market is highly fractured, with ~15 prequalified manufacturers and 140-210 (75% confidence interval) manufacturers of non-prequalified ACTs globally. Some evidence suggests that non-prequalified ACTs are widely sold.
  • We spent ~1.5 hours looking but have not found any breakdown of volume by manufacturer/site. Online research and expert conversations have not indicated that certain brands have a higher prevalence of S&F drugs.
• We are unsure about the relative supply and demand for ACTs. It is possible that actions to limit the supply of S&F drugs (especially if ~85% of S&F drugs have < 100% API but remain somewhat effective, as we estimate in Table 1) could lead to shortages and more untreated cases of malaria.
  • Rough calculations suggest that stockouts in the public sector might increase the probability of buying S&F drugs by 15-20% (70% confidence interval), with additional risks of increased price of ACTs in the private sector and non-treatment of the poorest patients.

Detail:

Most S&F ACTs (estimate: 65%-85%) are probably being sold in the private sector, which is not likely to affect SSA

Our thinking to get to the top-line estimate of where more S&F ACTs are being sold is as follows:

• We estimate that roughly 55% of ACTs are being sold through the public sector in SSA, based on:
  • A Unitaid (2018) forecast of ACT demand projected that 589M courses of ACTs would be needed in Africa in 2021, with 64% sold in the public sector.
  • Large-scale representative surveys³⁵ from 2015-2019 suggest that an average of 71% of under-fives (U5s) are treated in the public sector (WHO, 2021, p. 75). However, this figure does not seem to be population-weighted (WHO, 2021, p. 124), and plotting the most recent survey data for each country in SSA, where this is available, we can see there is significant variation (see Figure 5 below). While the average is high, the largest and most malarious countries in SSA (Nigeria and the DRC) show <50% public sector treatment.
  • The WHO estimates that ~200 million treatment courses of ACTs are delivered each year through national malaria programs (WHO, 2021, p. 74), which is lower than the Unitaid forecast, suggesting public volumes may be lower.³⁶

Figure 5: Source of advice or treatment for U5s with fever, where treatment sought

Note. Most recent DHS and MIS results, sub-Saharan Africa only.

• We expect that the majority of ACTs in the public sector are pre-qualified and, therefore, have low S&F prevalence, but we still expect that S&F drugs can enter this sector.
  • Donor procurement processes often require purchase from WHO pre-qualified manufacturers, e.g., see the Global Fund policy here. These manufacturers (and specific manufacturing sites) have been inspected and found to pass certain quality standards. The budget submissions from Nigeria and the Democratic Republic of Congo to the Global Fund suggest that 75%-100% of ACTs in the public sector are donor-funded (see Table 2 below).
    • We sense-checked whether it is true that most ACTs in the public sector are donor-funded and high quality. Speaking to Deo Cibinda, a doctor in the DRC, he confirmed that the DRC strategy is to provide antimalarial drugs for free to all public health facilities. Donor-funded ACTs (e.g., from Global Fund or USAID) received at these facilities are high quality, but there are many stockouts.
    • We think the DRC strategy referenced above reduces the likelihood that ACTs in the public sector will be S&F, and we would expect that any S&F ACTs are more likely to be substandard rather than falsified.
  • We do not expect the prevalence in the public sector to be zero because S&F drugs can still enter this supply chain. For example:
    • It seems that Ministries of Health might purchase ACTs that are registered locally but not prequalified, e.g., looking at the list of registered drug products with the Nigeria National Regulatory Authority (NRA) in February 2018 (here),³⁷ there appear to be some manufacturers that are not WHO-PQ listed.
    • Michael Deats highlighted that even if a hospital or health facility is sourcing from a licensed wholesaler, it is dependent on the wholesaler sourcing from a safe supplier.
      • Similarly, Deo Cibinda shared that when there are stockouts in the public sector, a clinic may need to purchase ACTs directly from a private pharmacy.

Table 2: Breakdown of ACTs by sector, from three sources

	Unitaid forecasts for Africa (2021)38		Democratic Republic of Congo (2022)39		Nigeria (2022)40
Sector	Volume (% of total)	% pre -qualified	Volume	% pre- qualified	Volume	% pre- qualified	Notes
Public - donor funded	374.8m (63.6%)	100%	23.1m	100%	35.2m	100%	Expect lower prevalence for pre-qualified drugs, and donors (e.g., GF/USAID) have stronger procurement policies and testing. Experts: S&F can enter the public sector if goods were purchased from private (even if licensed wholesaler who has bought S&F products); or if a clinic needs to buy from the market during stockouts.
Public - nationally funded			0m	-	11m	Unknown
Private formal	74.1m (12.6%)	47%	Unknown	Unknown	60.7m Note: this seems like an over- estimate41	Unknown	Experts: pressure for pharmacies to compete with the informal sector leads to purchase of lower price drugs, which are more likely to be S&F.
Private informal	140.6m (23.8%)	27%	Unknown				Prevalence expected to be above average

• The above information would suggest that the private sector (45% of all ACTs) has a higher than average prevalence of S&F drugs. There is some evidence to support this suggestion.
  • Antimalarials are often available “over the counter,” e.g., in DRC, Nigeria, and Ghana (WHO, 2019, p. 19).
    • Pharmacists may not have been trained on how to identify S&F drugs. A desk review of the curriculum for six SSA and two Asian states found only one contained specific mention of training on S&F drugs (Ferrario et al., 2019, p. 2).
  • Testing of samples in Uganda by Nyqvist et al. (2022) showed that 37% of all drug stores were selling substandard or falsified ACTs (as identified using handheld spectroscopy).
  • One survey with a small sample (predominantly in Nigeria) suggests that almost 60% of respondents who received counterfeit drugs received them in pharmacies, while 7% received them from doctors (Onuh et al., 2022). We would not put much weight on this evidence.⁴²
• Taking this information together, we constructed low-confidence estimates of prevalence by sector, as we did not find existing estimates in the literature.
  • We first assumed an average prevalence of ~15%.⁴³
  • We then modeled three scenarios for the public sector, assuming a lower-than-average but non-zero prevalence of S&F drugs, which may differ based on whether drugs are purchased by donors or not. These scenarios are:
    • Low prevalence (one-third of average prevalence)
    • High prevalence (two-thirds of average prevalence)
    • Differentiated prevalence by source of drugs in the public sector, with a much lower prevalence for donor-funded supplies⁴⁴
  • In each scenario, the prevalence in the private sector was then back-calculated to achieve our expected weighted average prevalence of ~15%.
• We can use these scenarios to get a sense of how prevalence might plausibly vary from different sources. They are unlikely to be representative, especially given the variation in countries across SSA, but making these rough calculations allows us to:
  • Estimate a range for the proportion of all S&F drugs that are found in each sector (as shown in Table 3 below)
  • Estimate how much the likelihood of buying S&F ACTs increases when there are stockouts in the public sector (more here)

Table 3: Modeled scenarios of S&F prevalence by sector

		Scenario 1: Low prevalence in public sector		Scenario 2: High prevalence in public sector		Scenario 3: Differentiated prevalence in public sector
Sector	Market share	Prevalence (%)	% all S&F drugs	% prevalence	% all S&F drugs	% prevalence	% all S&F drugs
Public - donor funded	44%	5%	15%	10%	29%	2%	6%
Public - nationally funded	11%	5%	4%	10%	7%	14%	10%
Private	45%	27%	82%	21%	63%	28%	84%
Weighted total		14.9%		15.0%		15.0%

Note. Market share is calculated using the private/public sector split of 45%/55% estimated in previous sections, and the public sector proportion is further split between donor- and nationally-funded ACTs based on the ratio for DRC and Nigeria combined, see Table 2.

We are uncertain about procurement processes for the public sector. However, based on first principles, it seems likely that tractability will remain somewhat limited due to reliance on the same national institutions.

• Figure 6 below from Amadi & Tsui (2019) describes the procurement and distribution systems in Nigeria. Overall, the public and private processes look to involve similar players (with the exception of central medical stores).
• The above phenomenon suggests that addressing S&F drugs in the public sector may encounter similar issues of tractability as discussed in the previous section.

Figure 6: Interactions of key players in the procurement and distribution process

Note. From Amadi & Tsui ( 2019 ).

The ACT market is highly fractured, with ~15 prequalified manufacturers and 140-210 (75% confidence interval) manufacturers of non-prequalified ACTs globally

• Based on the WHO list of prequalified finished pharmaceutical products for malaria, there appear to be ~15 different companies operating across ~30 different manufacturing sites.⁴⁵ These sites are mostly in India and China, while sites in Africa appear to be only in Morocco and Kenya.
• We estimate that roughly 140-210 non-prequalified manufacturers of ACTs (75% confidence), some of which have sites in SSA, are commonly found in the region. As mentioned above, these manufacturers can be registered by NRAs without being prequalified.
  • A study of more than 330,000 antimalarials available in eight SSA countries between 2009 and 2015 identified 185 manufacturers versus the twelve that were on the prequalified list at the time (Nayyar et al., 2019, p. 1059).
    • A smaller and more recent study (data collection in 2016-2018), identified 124 brands being sold in Kinshasa in the DRC; 44% of brands were not registered, and a further 14.5% had expired registrations (Landu et al., 2019).
  • Some information found in a brief search on Nigeria suggests locally made, registered ACTs are commonly found. In de Haan et al. (2021), a malaria policymaker for the Nigerian government is quoted as saying: “The local manufacturers produce all the ACTs, but the barrier is that none of them is WHO pre-qualified.”
    • It is unlikely that locally made, registered ACTs are common across all of SSA. Other countries are less likely to have domestic production (e.g., de Haan et al. [2021] note that Burkina Faso does not), and other countries may have more tightly regulated private sectors, e.g., Rwanda.
• We spent ~1.5 hours looking, but we have not found any breakdown of volume by brand or manufacturer/site. Online research did not reveal any data that indicates that a higher prevalence of S&F drugs is associated with certain brands of ACTs, and Paul Newton (Head of the Medicine Quality Research Group at the University of Oxford) and Deo Cibinda did not identify any particularly poorly performing brands.
  • We think this information would be valuable and have highlighted it along with other research we would like to see here.

We are unsure about the relative supply and demand for ACTs, and we are unsure whether actions to limit the supply of substandard drugs could lead to more untreated cases of malaria

• We spent ~30 minutes looking for an overview of the ACT market that would give a sense of relative supply and demand. Shreta and Yadav (2012) capture the market dynamics in a period in the past when the rapid growth of demand outstripped supply, but it is unclear how closely these correlate to supply and demand now and what capacity existing manufacturers have to scale.
• There is some risk that actions that shut down producers of substandard drugs may result in shortages and untreated cases of malaria. This phenomenon is particularly salient if ~85% of S&F drugs have < 100% API but remain somewhat effective, as we have estimated in Table 1.
• Rough calculations suggest that stockouts in the public sector might increase the probability of buying S&F drugs by 15-20% (70% confidence interval), with additional risks of increased price of ACTs in the private sector and non-treatment of the poorest patients.
• Referring back to our modeled estimates of prevalence per sector in Table 3, the difference in the likelihood of purchasing S&F drugs in the private sector versus the public sector is 22% in Scenario 1, 11% in Scenario 2, and 23.6% in Scenario 3.⁴⁶ A 70% confidence interval of 15-20% increased likelihood of purchasing S&F drugs seems reasonable.
  • Note that this likelihood may be very different across countries and depending on whether a consumer visits a formal part of the private sector (e.g., a pharmacy, which may or may not be licensed under a government scheme) or the informal market.
• A differences-in-differences study by Fitzpatrick (2022) suggests that in addition to a higher likelihood of purchasing S&F drugs, a patient trying to buy ACTs in the private sector during a stockout is likely to face higher prices (estimated 35% increase).
  • The same paper suggests that poorer and less educated individuals may not receive treatment at all.

A shallow assessment doesn’t point towards a single promising solution, but we have highlighted those that seem most interesting

Top-line findings:

• This table shows our assessment of 24 potential interventions, including our thoughts on tractability and some information about who is currently working on this and whether there is room for more funding.
  • The Bill and Melinda Gates Foundation (BMGF) seems to be the largest funder. Other key actors include WHO and United States Pharmacopeia (USP), which is associated with USAID. It seems that countries are using Global Fund (GF) allocations to strengthen national labs.
  • A low-confidence estimate of total philanthropic spending is $30M-$50M per year, excluding GF spending on national labs and non-medicines WHO-PQ budget.
• Overall, we did not find any intervention that appeared to be a silver bullet; most were limited by tractability — a need for high state capacity and political will in SSA or manufacturing countries like India and China.
• We defined four stages for possible interventions, and the most promising from each stage (in decreasing order) is:
  • Limit purchase of S&F ACTs by consumers: The most tractable way to do this may be to fund subsidized high-quality ACTs in the public sector in order to crowd out S&F ACTs and reduce the overall cost of drugs. There is some evidence that this strategy has worked previously and could be pursued at a relatively low cost. More here.
  • Data collection to inform interventions: BMGF is funding a three-year study to determine the prevalence of S&F antimalarials in Benin, Cameroon, and Nigeria. It may be possible to expand this study (geographically/in terms of data collected) or to leverage the data for advocacy. More here.
  • Limit entry of S&F ACTs into LMICs: The most shovel-ready option would be to fund the Promoting the Quality of Medicines program (PQM+) to support countries in strengthening supply chains/national capacity. Budgets suggest support tends to cost <$0.5m per year, but we have not found any evaluations of cost-effectiveness. More here.
  • Reduce manufacturing of S&F drugs at source: The most tractable strategy seems to be providing technical assistance to manufacturers to improve quality, which would likely need to be motivated by volume commitments.
    • Interventions to bolster WHO-PQ may be limited by room for more funding, as outlined in our previous report on WHO-PQ.
• We also created a rank-ordered research wish list for this area; see here.

Detail:

• We found only one review of the effectiveness of interventions, which was based on limited data and could only evaluate two interventions: El-Jardali et al. (2015). Licensing pharmacies was not found to be effective, but registration of products was.
• Major actors in this field appear to be:⁴⁷
  • Bill and Melinda Gates Foundation — focusing mostly on international and regional initiatives to support regulation and registration (African Medicines Agency, WHO-PQ), and some funding for data collection and reporting (e.g., $3.5m for prevalence studies).
    • Together with the EU (both European Commission and specific countries), have committed to mobilize >€100M over the next five years for regional and national initiatives; see here.
  • United States Pharmacopeia (USP) — working in countries supported by USAID. Promoting the Quality of Medicines program (PQM+) has a broad scope, its relevant areas of focus appear to be strengthening NRAs and national labs, and helping countries develop risk-based post-market surveillance. See more in Appendix F.
  • WHO — focusing on strengthening NRAs (in LMICs and manufacturing countries) and running WHO-PQ, as well as dedicating time to raising awareness with pharmacies and consumers.
  • Global Fund — it seems that countries are using their allocated funds to strengthen national labs, though it is difficult to know how much of these funds go towards testing medicines to identify S&F drugs.
• Many of the interventions to address the private sector rely on the presence of a strong national regulatory authority (NRA), as well as customs and police, and political will to address the issue (see table).
  • Michael Deats highlighted that close working between the NRA, customs, and police is necessary but often lacking. Political culture is often not supportive of the necessary transparency to quantify and address the issue (also highlighted by Paul Newton).
  • The WHO is in the process of reviewing NRAs using a global benchmarking tool, which is used to define the organization’s maturity level on a scale of 1-4, with 4 being the best.
    • In Africa, as of November 2022, only Ghana, Nigeria, South Africa, and Tanzania have achieved Level 3, indicating that they are “stable, well-functioning and integrated” (WHO, 2022a; WHO, 2022b). Note that not all countries in SSA have been assessed.
  • Hamilton et al. (2016) highlight Rwanda’s success in tackling this issue, though it is important to note that Rwanda is a country with high levels of state capacity.
• There are some options that require lower levels of state capacity and may be more tractable. We highlighted four options above and provide more detail here.
• Crowding out non-prequalified medicines in the private sector: we are aware of two approaches that have been attempted and evaluated for ACTs.
  • Provision of subsidized ACTs in the community by an NGO: This option is the Living Goods model. An RCT was conducted in Uganda to assess this strategy in 2010/11, and suggested that this intervention reduced the number of private stores selling fake ACTs by 46% (after controls) and reduced the average price of ACTs by 16% (Nyqvist et al., 2022).⁴⁸
    • This intervention would involve subsidizing ACTs so that they can be sold for 20-30% below market price. Based on one source at an online pharmacy in Nigeria (see here), Coartem costs ~$4.40 online, so a 25% subsidy would be roughly $1.10 per treatment with a subsidized price of ~$3.30.
      • However, we would expect the actual subsidy to be < $1.10, as it is likely possible to procure the drug for less than $4.40. For example, Global Fund pooled procurement prices here suggest that they pay up to $0.63 for an adult treatment course of artemether lumefantrine.
    • GiveWell assessed Living Goods in 2014 (see here) and designated the organization as a standout charity. Their evaluation was limitedly focused on drug quality (given lack of data) and is likely out of date (given changes at Living Goods). An RCT was conducted to re-evaluate the intervention in Uganda between 2016 and 2021, and the study includes assessments of drug quality (AEA RCT Registry, 2020). At the time of writing, the results of the study were unpublished. GiveWell plans to re-evaluate this intervention once the RCT is completed. See more notes in Appendix F.
  • Co-payments by donors to supply subsidized ACTs to pharmacies: Donors originally established the Affordable Medicines Facility–malaria (AMFm) from 2010-2013 in seven countries in SSA, achieving lower prices for ACTs through negotiation and direct co-payments by donors to manufacturers.
    • The aim of this intervention was to shift markets away from artemisinin monotherapies and from non-quality assured ACTs. There is some evidence that the program achieved the first goal but that the shift to quality assured ACTs somewhat reversed when the subsidies ended (see Appendix F).
    • Following AMFm, the Global Fund set up a co-payment mechanism that makes it technically possible for the seven participating countries to use their allocation to continue to subsidize private-sector ACTs, but according to WHO (2019, p. 23), since “2017, the CPM has been terminated in Nigeria and is being reduced significantly in Kenya and Uganda.”
    • Logistically, it might be possible for a donor to provide funds to a Ministry of Health for another activity currently covered by the Global Fund allocation, to free up this money to be used for co-payments. If possible, such a donation should be relatively scalable.
• Extending or leveraging the BMGF prevalence studies: BMGF made a $3.5m grant to Population Services International to monitor the prevalence of S&F drugs in Benin, Cameroon, and Nigeria for three years. Possibly impactful extensions of this work could be:
  • Funding expansion to more countries: As mentioned above, much of the available data is already from Nigeria (and Tanzania and Ghana).
  • Funding enhanced data collection: It may be that this program could also conduct some of the research that we have highlighted as valuable for marginal additional cost.
  • More advocacy based on these results: Additional funding could also support advocacy, similar to the LEEP model of sharing results with national bodies to encourage action. (It seems likely that advocacy is part of BMGF’s project in the selected countries.)
• Strengthening supply chains/national agencies through the PQM+ program: The Promoting the Quality of Medicines program (PQM+) is run by United States Pharmacopeia (USP) and funded by USAID.
  • Their work spans multiple stages in our table of potential interventions; see more examples in Appendix F.
  • Our sense is that the program works on a consultancy-based model, where country programs may choose to allocate some of their USAID budget to finance a project with USP.
    • If USP are operating as a consultancy, it may be that they could be approached with donor funds to support either a) additional projects in USAID-funded countries or b) new projects in non-USAID countries.
    • Alternatively, a donor could conduct advocacy to encourage USAID countries to use their USAID budget for these kinds of projects.
  • Based on several budgets for the President’s Malaria Initiative (PMI), three examples suggest that PQM+ support generally costs < $0.5M per year. However, we have not seen any evaluations of cost-effectiveness.
    • Nigeria 2020, $0.5M: to “Strengthen local pharmaceutical manufacturers to produce quality antimalarials. Continue support for NAFDAC's capacity for drug quality control including the procurement of necessary equipment and supplies. Support will include deployment and use of minilabs for field testing of drugs. Activities include post market surveillance in priority states to detect counterfeit and poor quality medicines” (PMI, n.d., p. 6).
    • Mali 2022, $0.25M: to “[s]trengthen national laboratory, including quality assurance and quality control of antimalarial drugs. Continue supporting sentinel sites and sample review” (PMI, n.d., p. 5).
    • DRC 2022, $0.3M: for “[p]ost-marketing surveillance of antimalarial

      medicines and technical assistance… for quality assurance and quality control of antimalarial medicines” in nine states (PMI, n.d., p. 5).
• Reduce manufacturing of S&F drugs at the source by improving processes: A brief search (<30 minutes) identified two organizations that could be well placed to do this and there are likely others.
  • The first organization is the PQM+ program, which is already working on reducing the manufacturing of these drugs in some countries (e.g., Myanmar). The second organization is QUAMED, a non-profit that provides technical assistance and, in 2021, worked with clients such as FHI360, UNHCR, and the French Red Cross. QUAMED appears to be relatively small (with a turnover of $0.3M).⁴⁹
  • Based on first principles, it seems that technical assistance could lead to better production and a lower prevalence of S&F antimalarials. However, it is unclear how much of a reduction there would be and how long any improvement would last.
    • It also seems likely that profit-driven manufacturers would need to be incentivized to take part (e.g., through volume commitments), and possible that improvements in manufacturing could be reflected in increased costs and, ultimately, increased prices.
  • Interventions to bolster WHO-PQ may be limited by room for more funding. We think it is unlikely that the program could absorb $10M-$30M per year to address ACTs only, given an estimated annual budget of $30M-$40M across medicines, vaccines, vector control, and diagnostics. It is also unclear whether WHO-PQ is presently able to increase their staff, as outlined in our previous report on WHO-PQ.

What we would do with more time

We believe the following actions would round out our report, though they may not change our conclusions:

• Estimate the health burden of S&F antihypertensives and briefly review whether the tractability of interventions could plausibly be higher for S&F antihypertensives than for S&F antimalarials.
• Get a sense of to what extent the problem of S&F drugs is likely to get worse in the future, incorporating factors such as:
  • Increasing use of online pharmacies
  • Production of bio-synthetic artemisinin
  • Farming of artemisia in Nigeria
  • Roll out of the malaria vaccine
  • Increasingly globalized markets
• Speak with someone at PQM+ to better understand their ongoing work, the successes they’ve had, and whether our first principles assessments of tractability are reasonable
• More thoroughly review the use of mobile authentication systems in Nigeria to understand to what extent poor impact may be attributable to implementation versus technology versus consumer behavior
• Review other authentication methods using a report from WHO (2017) as a starting point (p. 15 onwards)

Research/data we would like to see

We highlighted above that we think S&F antihypertensives may have a similar burden to S&F antimalarials. It may be worth conducting a shallow level of research to:

1. Turn our intuitive thinking on burden into a back-of-the-envelope calculation.
2. Reconsider the tractability of potential interventions (summarized here) for antihypertensives. These drugs are used more globally than antimalarials, and countries outside of SSA may have higher state capacity to support interventions.

On the question of S&F antimalarials specifically, our prioritized list of research/data we would like to see is as below. As tractability is currently our main concern about this topic, items that would help us better understand interventions appear closer to the top of the list.

1. A prevalence study that focuses on collecting information that will help us better understand what interventions could be tractable. In order of priority:
   1. A volume breakdown of ACTs available to consumers by sector (public, private), brand/manufacturer, and manufacturing site.⁵⁰
   2. Information about other quality attributes of drugs, e.g., dissolution failures, degradation, and (toxic) impurities to help pinpoint where in the drug supply chain interventions might be most promising (e.g., manufacturing vs. storage vs. transportation).
   3. Granular information about API so that we can link this to treatment efficacy. Ideally, this information would be more detailed than what is currently available in Ozawa et al. (2022) (e.g., % of drugs with API < 20%/30%/40%/…).
   4. Specific inclusion of underrepresented countries, as most prevalence estimates of S&F drugs come from only three countries in SSA.
2. A summary of the budgets, resources, responsibilities, and funding sources of national regulatory authorities.
3. Scientific research to develop low-cost, accurate, portable testing devices. Our understanding from speaking with Harparkash Kaur is that quantitative content analyses to measure the stated API in a formulation can only be carried out in a laboratory using specialist analytical equipment.
4. Modeling studies estimating the probability of widespread artemisinin resistance in SSA over the next decades and the estimated contribution of S&F antimalarials to this scenario would give more rigor to our health burden estimates.

Acknowledgments

Aisling Leow and Jenny Kudymowa jointly researched and wrote this report. Aisling Leow also served as the project lead. Tom Hird supervised this report. Thanks to Tom Hird and Melanie Basnak for helpful comments on drafts. Further thanks to Deo Cibinda, Michael Deats (University of Oxford), Harparkash Kaur (London School of Hygiene & Tropical Medicine), Andrew Moran (Resolve to Save Lives), Paul Newton (University of Oxford), and Nicholas White (University of Oxford) for taking the time to speak with us. Open Philanthropy provided funding for this report, but it does not necessarily endorse our conclusions.

We invite you to explore more RP research via our database and stay updated on new work by subscribing to our newsletter.

Appendices

Appendix A. Efficacy of antimalarials by percentage of API present

Figure A1: Estimates of efficacy of ACT and non-ACT medicines by % API category

Note. From WHO (2017).

Appendix B. Excess annual mortality due to artemisinin resistance

Figure A2: Excess mortality due to artemisinin and artemisinin-combination therapy resistance in malaria-endemic areas

Note. From Lubell et al. (2014).

Appendix C. Death and DALY burden estimates of substandard and falsified antimalarials

Table C1: Death and DALY burden estimates of S&F antimalarials

Study	Geographical scope	Estimated death burden	Estimated DALY burden	Comments & own calculations
WHO (2017)	Sub-Saharan Africa	230-529 deaths per 1M malaria cases seeking treatment (~2%-5% of all malaria deaths)	7k-15k DALYs per 1M malaria cases seeking treatment	Own calculation: Assuming 234M malaria cases in SSA and 67% of cases seeking treatment => 36k-83k total S&F antimalarial deaths and 1.1M-2.4M51 DALYs in SSA
Renschler et al. (2015)	Sub-Saharan Africa	122k under-5 deaths (~4% of all under-5 deaths [not only malaria deaths]) Most deaths in Nigeria (61%), Uganda (8%), and DRC (5%)	N/A	Own calculation: Assuming that 76% of malaria deaths in SSA occur in children below 552 => ~160k total S&F antimalarial deaths
SAFARI model—Ozawa et al, (2019a)	Uganda	1.1k under-5 deaths (~9% of all under-5 malaria deaths)	79k under-5 DALYs ~7% of total under-5 malaria DALYs	Own calculation: Assuming total malaria DALYs in SSA of 43M and 7% of this due to S&F drugs => 3M53 total S&F antimalarial DALYs in SSA
SAFARI model—Jackson et al. (2020)	Zambia	213 under-5 deaths (8% of all under-5 malaria deaths)	N/A
SAFARI model—Beargie et al. (2019)	Nigeria	12k under-5 deaths (16% of all under-5 malaria deaths)	N/A
SAFARI model—Ozawa et al. (2019b)	Democratic Republic of the Congo (DRC) - Kinshasa and Katanga regions	10.5k under-5 deaths (33%-43% of all under-5 malaria deaths, depending on region)	N/A

Note. None of the models in this table account for antimicrobial resistance.

Appendix D. Death and DALY burden estimates of widespread artemisinin resistance

Table D1: Death and DALY burden estimates of widespread artemisinin resistance

Study	Geographical scope	Estimated death burden	Estimated DALY burden	Comments & own calculations
Lubell et al. (2014)	Sub-Saharan Africa	~105k under-5 deaths (~30k-300k deaths in sensitivity analysis) See Appendix B for geographical spread of deaths	N/A	Own calculation: Assuming that 105k under-5 deaths represents 76%54 of all malaria deaths => total of 138k deaths due to S&F antimalarials in SSA (~40k - 400k in sensitivity analysis)
SAFARI model—Ozawa et al. (2019a)	Uganda	~900 under-5 deaths (7% of all under-5 malaria deaths)	63k under-5 DALYs (6% of all under-5 malaria DALYs)	Own calculation: Assuming we could extrapolate 7% of all deaths and 6% of all DALYs to SSA55 => total of 42k deaths and 2.6M DALYs due to S&F antimalarials in SSA
Prof. Nicholas White (expert opinion)	Sub-Saharan Africa	~400k deaths56	N/A

Appendix E. Drivers of artemisinin resistance

Figure E1: Drivers of artemisinin resistance

Note. From WHO (2022b, p. 14)

Appendix F. Notes to accompany list of interventions

In no particular order.

Mobile authentication systems:

• Nigeria’s National Agency for Food and Drug Administration (NAFDAC) introduced a nationwide campaign in 2012 to cover all goods, but especially antibiotics, anti-inflammatories, and antimalarials. If consumers detected S&F products, they were directed to report all cases.
• This system “uses scratch codes and Short Messaging Service (SMS) to empower consumers to verify the authenticity of medicines at the point of purchase. The consumer scratches a panel on the product which reveals a unique, one-time use PIN which is sent toll- free to a short code using any of the GSM operators and the consumer receives a response in form of a text message (SMS) stating that the product is either genuine or suspected fake” (NAFDAC 2018, p. 1).
  • A number of different systems have been around for a while: “mPedigree (Nigeria) in 2006 (Wall, 2017), Sproxil (Nigeria) in 2009 (Wall, 2017), İlaç Takip Sistemi (Turkey) in 2010, and Pharmsecure (Nigeria and India) in 2012” (Rasheed et al., 2018).
• NAFDAC (2018, p. 10) suggests that all funding was coordinated by NAFDAC.
• Skimming a number of papers⁵⁷ suggests that awareness was ~65%, and poor connectivity by network services reduced the effectiveness of the system. Other shortcomings included NAFDAC failing to update the central database leading to false negative reports (Amadi & Tsui, 2019, p. 1351).
• We did not find any evaluations of the impact on S&F drugs.
• Michael Deats shared that he has seen cases where this system is cheated by those producing S&F drugs, e.g., they set up a fake number to SMS/call, which automatically responds that the product is okay. He is also skeptical that levels of use are high in SSA.

Other authentication systems:This list from WHO (p. 15 onwards)

Crowding out S&F drugs in the private sector — NGO provision:

Notes on key paper: Nyqvist et al. (2022).

• An earlier version of the paper accessible from JPAL can be found here.
• An RCT conducted in 4 districts in Uganda in 2010/11, in cooperation with Living Goods.
  • In total, 49 villages received the intervention and 50 villages were in the control group.
  • An NGO purchased high quality ACTs, and employed one CHW in each village to conduct household visits and sell on for 20-30% below market price.
    • The NGO was subsidizing the price by 30-40%, with the CHW in each village keeping a 10% commission.
  • Baseline survey in early 2010, and follow-up approximately 18 months later in a subset of 48 of the 99 original villages.
• The intervention reduced the number of private stores selling fake ACTs by 46% (after controls) due to a combination of exiting from the market (entirely), stopping the selling of ACTs, or switching to selling quality ACTs. A back-of-the-envelope calculation by the authors demonstrates that exiting accounts for roughly half of the effect.
  • At the same time, average fall in price: 16%.

Notes on GiveWell’s assessment:

• Living Goods is still focused on selling subsidized high-quality ACTs and other essential medicines:
  • “The CHPs [...] also sell preventive and curative health products. The product line they have at disposal includes [...] curative treatments (e.g., oral rehydration salts, zinc, and ACTs) [...]. These products are sold by the CHP at a discount. The retail price is determined by the NGOs head office with a target of keeping prices for preventive and curative products about 20% lower than the prevailing local market prices” (Evaluation of Living Goods/BRAC entrepreneurial CHW model in Uganda - Phase II - AEA RCT Registry, 2020).
• A currently ongoing RCT will also evaluate impacts on drug quality and S&F drugs (AEA RCT Registry, 2020).
• GiveWell evaluated Living Goods in 2014 and deemed it a “standout charity” (before they discontinued the “standout charity” designation in 2021):
  • GW’s assessment included a preliminary assessment of Living Goods’ impact on drug quality. However, they did not seem to put much weight on this and evaluated Living Goods at an early stage before detailed, high-quality monitoring data was available and before Living Goods scaled massively. Moreover, since then, Living Goods slightly adjusted its approach w.r.t. ensuring drug quality, which was not accounted for by GW yet.
• GW is planning to update its assessment once the currently ongoing large-scale RCT (AEA RCT Registry, 2020) is published.

Crowding out S&F drugs in private sector — subsidies/co-payments:

• Two mechanisms that exist/were used in the past:
  • Affordable Medicines Facility–malaria (AMFm) ran from 2010-2013: a program which negotiated lower prices and gave manufacturer-level subsidies accompanied by behavior change campaigns. This program ran in seven countries in SSA: Ghana, Kenya, Madagascar, Niger, Nigeria, Uganda, and Tanzania (mainland and Zanzibar).
  • GF Co-Payment Mechanism (CPM): the successor to the AMFm program, available in six of the countries from the AMFm pilot. Countries can choose to use some of their GF funding to subsidize ACTs. There is more detail in the Global Fund Operational Policy Manual (2022, p. 120).
    • However, according to WHO (2019, p. 23), since “2017, the CPM has been terminated in Nigeria and is being reduced significantly in Kenya and Uganda.”
  • The assessment of baseline and endline (Figure F1 below, found in WHO, 2019, p. 23) shows a switch towards QAACTs in most countries. However, more recent data from ACTwatch + CHAI (Figure F2 below, found in WHO, 2019, p. 24) suggests this was somewhat reversed as co-payments/subsidies were reduced.
• More evaluation here that the authors did not have time to read.

Figure F1: QAACT market share

Note. From WHO (2019, p. 23).

Figure F2: Market share of QAACTs, non-QAACTs, and non-ACTs (all shops)

Note. From WHO (2019, p. 2).

PQM+ program:

• This program was funded by USAID for the period 2019-2024, following the original PQM program (2009-2019). Our sense is that the program works more on a consultancy-based model, where country programs may choose to allocate some of their USAID budget to finance a project with USP (example here). We didn’t find anything to indicate whether they have a central budget for overheads from USAID.
• This program has an overall emphasis on systems strengthening to promote quality medicines, with five overarching goals (see here). Most relevant work includes:
  • Strengthening links between NRAs and other government agencies, and supporting transparency initiatives
  • Helping to develop and implement guidelines/regulations
  • Training and capacity building in NRAs and labs
  • Development of cost-effective risk-based post-market surveillance
  • Modeling prevalence and impact of S&F drugs⁵⁸
• Some examples of what they have been working on, based on their Q2 report for 2022 (here):
  • “[E]stablishing systems to promote transparency and collaboration between key stakeholders (notably in Guinea, Kenya, Liberia, Nigeria, and Mozambique)” (pp. 2-3)
  • Risk-based post-marketing surveillance in 18 countries, including DRC, Ethiopia, Ghana, Guinea, Kenya, Nigeria, Rwanda (p. 3)
  • Providing “technical assistance to the Liberia Medicines and Health Products Regulatory Authority (LMHRA) to expedite and clear its backlog of 52 dossiers on antimalarial medicines” (p. 4)
  • Providing support to 12 African national quality control labs, including “guided supervision of quality management system (QMS) document development, trainings, and support for pre-assessments by two countries’ regional accreditation bodies” (p. 4)
  • “[D]evelop[ing] a model strategic plan for NMRAs to operationalize their institutional development plans derived from country Global Benchmarking Tool (GBT) assessments” (p. 8)
  • Providing technical assistance to the only manufacturer of antimalarials in Myanmar, creating a roadmap for certification and conducting audits (pp. 44-45)

African Medicines Regulatory Harmonization (AMRH) program:

• This program was originally founded in 2009, and funded by the World Bank between 2011 and 2021.
  • Mercado et al. (2022, p. 3) suggests donor funding of ~$3.5m per year: “Starting with an initial contribution of US$12.5 million from the BMGF, the WBG set up the AMRH Trust Fund and quickly mobilized additional funding from other donors including the United Kingdom’s Foreign Commonwealth and Development Office, US Government/President’s Emergency Plan for AIDS Relief (PEPFAR), and GAVI, increasing its programmatic budget from US$12.5 million to about US$35.0 million over the last decade.”
  • This money was also used to fund regional initiatives, e.g., in the East African Community here
• The current website suggests this program is now under the auspices of the African Union Development Agency — New Partnership for Africa’s Development
  • It is not clear what the budget or funding situation is for this program. Since 2021, BMGF has funded some regional initiatives, e.g., $0.4m awarded in 2022.
  • There are eight technical committees.
• The African Medicines Agency also appears to be linked to the AMRH.

WHO Global Surveillance and Monitoring Systems:

• Based on WHO webpage here, focal points in countries submit reports of S&F products (they could be suspected or validated).
  • WHO compares that report to anything else in the database (can allow for links across countries) and provides technical support.
    • This support could be access to labs or investigation.
  • Speed of response depends on whether adverse reactions are indicated.
    • If “serious threat to public health” will issue a medical product alert.
• Medical product reports are very rare: as of December, there were a total of six in 2022.
  • It is unclear how many reports there are in the system that do not get to the stage of medical product report

Barcodes/GS1/End to end/Track and trace:

• Review of digital tools: Rasheed et al. (2018)
• USAID explanation of use and benefits here and poster on progress here

WHO country insight studies on consumer awareness campaigns:

• These campaigns are being conducted in Nigeria, Ghana, Sierra Leone, and Uganda (WHO, 2020, p. 5).
• Michael Deats shared that the awareness campaigns are run by WHO and then evaluated by a UK university.

Notes

---

1. Michael Deats shared that falsified goods can have many sources: a) fully imported products, b) imported high-quality packaging that is smuggled into a country and filled locally, c) locally produced (low-quality) packaging and filled locally, d) used packaging that is salvaged from rubbish and filled locally, or e) expired medicines with dates changed.  ↩

2. I.e., the proportion of drugs tested that were found to be S&F. ↩

3. This range refers to antimalarials in all regions were tests were performed, not only SSA. ↩

4. This range refers to all tested S&F drugs in SSA, not only antimalarials. The second highest prevalence is in Asia (10.2%-13.7%). ↩

5. The literature does not report a direct estimate of S&F antimalarial prevalence in SSA, but we expect that it is likely at the higher ends of the ranges we show here. ↩

6. “[T]his is often when constrained access to safe, affordable, quality products overlaps with poor governance systems, including unethical practice and corruption, and/or weak technical capacity for quality assurance during manufacturing and distribution” (WHO, 2017, p. 9).  ↩

7. Ozawa et al. (2022)’s S&F antihypertensive prevalence estimate is based on only six studies and ~3.6k samples. Their figure on S&F antimalarials (19.7%) is based on 34 studies and >19k samples. ↩

8. Other drug categories have an even higher estimated prevalence (e.g., genitourinary and sex hormone drugs: 56%; antiepileptics: 65% [WHO, 2017]), but the samples are even smaller and we expect their health burden to be lower. ↩

9. Do et al. (2021) reviewed the literature on S&F cardiovascular medicines and found an average prevalence of 15.4% of S&F medicines in 3,414 samples from 28 countries in four continents. Given that no data exists for many countries and the reviewed studies are likely not representative, we expect that the true average prevalence might be somewhat lower. ↩

10. Based on a quick back-of-the-envelope calculation, we estimate that roughly 15M individuals per year take S&F antimalarials in SSA: There are ~234M annual malaria cases in SSA (WHO, 2022, p. 18), of which ~67% are treated with antimalarials (WHO, 2022, p. 73), with a treatment adherence of ~65% (WHO, 2017, p. 50). Based on Ozawa et al. (2018), we think that ~15% of antimalarials are S&F. Multiplying all these figures leads us to an estimate of ~15M individuals taking S&F antimalarials in SSA each year.  ↩

11. This is the most common type of malaria in sub-Saharan Africa. ↩

12. “The role of the artemisinin compound is to reduce the number of parasites during the first 3 days of treatment (reduction of parasite biomass), while the role of the partner drug is to eliminate the remaining parasites (cure)” (WHO website, 2022). ↩

13. We have not been able to determine the exact scale of the dots in Figure 3 as Ozawa et al. (2018) do not report the number of samples tested per country. We think that there are at least 926 samples for Nigeria and 106 for Ethiopia, but it might be more (as some samples come from multicountry studies where it is not clear how many samples come from each respective country). ↩

14. Unfortunately, we have not found any good overview for antimalarials specifically. ↩

15. Some measure it by whether API is below or above some threshold (thresholds used vary; e.g., Ozawa et al. (2022) consider a drug “likely falsified” if the API is < 50%, while other studies, such as Hauk et al. (2021), focus on whether any evidence exists for intentional fraud (e.g., packaging or the drugs look different; incorrect labeling). ↩

16. According to WHO (2017, p. 6), studies commonly consider drugs with APIs below 85-95% or above 105-115% to be substandard). Some studies do other types of quality tests, e.g., testing for degradation or impurities (see McManus & Naughton, 2020).  ↩

17. Harparkash Kaur’s best guess based on her own research and experience is also that substandard drugs are more prevalent than falsified drugs. ↩

18. According to Nosten and White (2007, p. 185), “[t]here has been some investigation of the optimum dose with in-vivo dose response studies suggesting that at least 2 mg/kg/d for three days of artesunate is required for maximal effect.” ↩

19. According to Nicholas White, “[t]hese are average assessments—so in the average person, a 2 mg/kg dose of artesunate will perform almost as well as 4 mg/kg (assuming the partner drug dose is not reduced as well), but people vary—some people need more drug than others, either because they have high parasite burdens (major risk for selecting resistance) or because they have unusual pharmacokinetics. So, to ensure everyone gets enough, you need to have a wide margin—hence the current dosing. For fully artemisinin sensitive parasites, 1 mg/kg of artesunate gives about 50% of maximal effect.” According to Angus et al. (2002, p. 778), 50% of the maximum effect occurs at 1.6 mg/kg of artesunate. ↩

20. According to Harparkash Kaur, “in vitro dissolution testing offers a valuable prediction of the in vivo bioavailability and bioequivalence of tablets and capsules. Dissolution tests measure the amount of stated active pharmaceutical ingredient (SAPI) released into the dissolution media over time following detailed protocols (official monographs) set out for most drugs in pharmacopeias (e.g., European, British, USP, WHO International). The protocols outline the details of the test conditions (dissolution buffer/solvents, stirring speed, tolerance levels of the API, and temperature for the assay). Even if the quantity of SAPI in a medicine is within pharmacopeia’s tolerance limits for content, the amounts released, i.e., bioavailability, may be lower, thus giving poor dissolution characteristics. In addition, the coating of a drug can be so hard that it not only fails the dissolution test but will pass through the body when ingested without releasing any of its therapeutic API. The dissolution tests require sophisticated apparatus as well as specialized analytical equipment such as high-performance liquid chromatography (HPLC) and reference standards of the compounds being tested in order to calibrate the equipment, all of which is expensive.” ↩

21. According to Harparkash Kaur, dissolution studies are rarely carried out as they are tedious to implement. ↩

22. According to Harparkash Kaur, degradation can manifest in a reduced API concentration and can also release potentially toxic degradation products, but has so far not been shown to be toxic for only two antimalarials (artemether/lumefantrine and artesunate/amodiaquine). ↩

23. A study from Nigeria found that 25% antimalarials that were initially classified as having a reduced API concentration were, in fact, degraded (Hall et al., 2016, p. 999). ↩

24. Hall et al. (2016) found that “a quarter of ACTs purchased in Enugu, Nigeria (concurrent study), that would have been classified as substandard, were, in fact, degraded” and that “degradation was one of the major hitherto not recognized causes of drugs failing chemical content analysis.” ↩

25. We have not seen a clear-cut definition of what constitutes ‘widespread’ resistance. The studies we have seen use this term pretty vaguely. Lubell et al. (2014) assume that widespread resistance leads to “ACT clinical failure rates of 30% and where policy has reverted to quinine to manage severe malaria.” ↩

26. Note that we extrapolated some of the estimates from the literature with our own reasoning, as some estimates were either only provided at the country level, or only refer to under-5 deaths/DALYs (i.e., not considering the whole population). Details of our calculations and assumptions made are in Appendix D. ↩

27. We calculated these figures as follows (all figures in parentheses are ranges): (20-40%) * (10-30%) = (2-12%). Then: (~40k-400k deaths and 1M-5M DALYs) * (2-12%) = (~1k-50k deaths and 20k-600k DALYs). ↩

28. According to one of the authors, “[t]hese results are powerful reassurance that artemisinins can continue to be considered as extremely useful for treating malaria. Clinicians no longer need to be immediately concerned that using artemisinins could increase resistance, and the artemisinin drugs should remain an essential part of treatment pathways, whatever the delay in parasite clearances that are sometimes observed” (St. George’s University of London, 2022).  ↩

29. “There is no evidence that artemisinin partial resistance alone has resulted in an increase in malaria morbidity and mortality in the Greater Mekong subregion” (WHO, 2022). ↩

30. According to Kaur et al. (2016), for Southeast Asia, “the most important driver of artemisinin resistance may be the prolonged use (nearly 30 years) of artemisinin monotherapy in Southeast Asia.” ↩

31. Paul Newton is doubtful that one can extrapolate findings on resistance from the South East Asian context to SSA. ↩

32. Malaria case fatality rate estimates in the literature range from 0.45 % to 60% (Ozawa et al., 2019, p. 12). ↩

33. Open Philanthropy (2021): “We’re also switching our approach to be more consistent with GiveWell’s framework in how we translate deaths into DALYs. [...] But we can use their moral weights to derive a mapping of deaths to DALYs, by dividing GiveWell’s moral weight for each death by GiveWell’s moral weight for a year lived with disability (which is defined by WHO so as to be equivalent to a DALY). For example, GiveWell places 60% more weight on a child malaria death than on an adult death, and we can fairly straightforwardly interpret their process as counting an average of 32 DALYs per adult malaria death, so the GiveWell-based DALY model would implicitly count 32*160% = 51 DALYs for an under-5 malaria death.” ↩

34. WHO (2017, p. 53): “Discounted years of life lost (per death): 28.22 (under 5 years), 24.3 (over 5 years)”. ↩

35. Specifically, the Demographic and Health Surveys (DHS) and Malaria Indicator Surveys (MIS).  ↩

36. Based on Figure 7.7, averaging the values for 2019 and 2020. ↩

37. It was not possible to download the current list from their website. ↩

38. Based on Unitaid (2018), specifically total ACT demand in Table 4, and quality-assured ACT (QAACT) demand in Table 5. We apportioned the total of private sector QAACT demand (Table 5) between formal and informal sectors using the global ratio from Figure 6. We equate QAACTs with pre-qualified, following the ACTwatch Group et al. (2017) definition as pre-qualified, in compliance with Global Fund Quality Assurance policy (which is most often for goods to be pre-qualified), or approved by the European Medicines Agency.  ↩

39. Budgeted figures based on the national malaria program’s submission to the Global Fund, which can be downloaded from here (see “Funding Request Malaria - 2020 - en”). Specifically, the “COD_M_ProgGap_en” file and the “CM-treatment gap tables” tab.  ↩

40. Budgeted figures based on the national malaria program’s submission to the Global Fund, which can be downloaded from here (see “Funding Request Multicomponent - 2020 - en”). Specifically, the “NGA_Z_ProgGap_en” file and the “ACTs” tab.  ↩

41. The budget suggests ~100M cases per year, hence the need for 60.7M ACTs in the private sector, but WHO (2022, p. 210) reporting estimates a total of 65.4M cases per year (upper bound: 89M).  ↩

42. The remaining proportion are reported as “Other.” The survey was conducted online, and the majority of those surveyed have a tertiary degree, making the sample fairly unrepresentative of SSA more generally. Most respondents are from Nigeria, but we cannot see a breakdown of results by country.  ↩

43. Based on Ozawa et al. (2018), we think that ~15% of antimalarials are S&F, whichis a slight downward revision of Ozawa et al. (2018)’s estimate, as we expect there might be an upward bias due to the sampling.  ↩

44. Equivalent weighted prevalence is roughly 4.4%, which is similar to scenario 1, which assumes just under one-third of average prevalence.  ↩

45. Based on the WHO list as it was in November 2022. The downloaded data can be found here, and the original source for all finished pharmaceutical products prequalified by WHO can be found here (live link).  ↩

46. For scenario 3, we assume a weighted average of 4.4% (equal to [2% * 44% + 14% * 11%]/ 55%), as this is the likelihood for any given patient, and subtract this value from 28% to get 23.6%.  ↩

47. The Fleming Fund is also spending £4M on surveillance/data collection, field screening equipment, and reporting, but with more of a focus on antimicrobial resistance; see here. ↩

48. An earlier version of the paper is accessible from JPAL here. More detailed notes on the study can be found in Appendix F.  ↩

49. Andrew Moran shared in an email that Resolve to Save Lives is working with QUAMED on the issue of S&F antihypertensives. ↩

50. This could also be a standalone study, and in the public and private formal sectors, this could potentially come from existing administrative data.  ↩

51. 234M * 67% * (230 to 529) ≅ 36k to 83k deaths; 234M * 67% * (7k to 15k) ≅ 1.1M to 2.4M DALYs. See here for sources. ↩
52. See here for sources. ↩
53. 43M * 7% ≅ 3M. See here for sources. ↩
54. See here for sources. ↩
55. 600k deaths * 7% = 42k deaths; 43M DALYs * 6% = 2.6M DALYs. See here for sources. ↩
56. Nicholas White: “Health estimates for malaria are imprecise, but it is not unreasonable to think that if we lose the ACT partner drugs (as would likely follow artemisinin resistance) we could return to a million deaths a year.” As we currently have ~600k malaria deaths/year in SSA, this would imply an additional 400k deaths. ↩

57. Adekoya & Ekeh (2021a), Ekeh & Adekoya (2021b), Oyetunde et al. (2019) ↩

58. Partnering with Sachiko Ozawa, mentioned above in the report ↩