Data systems for malaria burden estimation: Challenges, initiatives, and opportunities

Editorial note

This report was commissioned by GiveWell and produced by Rethink Priorities from August to September 2023. We slightly revised the report for publication in May 2025. GiveWell does not necessarily endorse our conclusions, nor do the organizations represented by those who were interviewed.

The primary focus of the report is a landscape analysis to provide an overview of the major challenges, barriers, and improvement initiatives with regards to malaria-related data systems in sub-Saharan Africa. Our research involved reviewing the scientific and gray literature, and we spoke with three experts.

Since the time of writing, the malaria data systems landscape has evolved, particularly due to recent disruptions in United States Agency for International Development (USAID) funding. In early 2025, the U.S. government significantly reduced USAID’s budget, leading to the reduction or termination of several major initiatives, including the Demographic and Health Surveys (DHS) Program, which ended in February 2025. Many of the efforts described in this report, especially those focused on strengthening routine malaria data, list USAID as a major funder. While we have not systematically assessed the impact of these changes, it is likely that some programs have since been altered, scaled back, or discontinued. These developments are not reflected in the body of the report but are noted here to provide context.

We don’t intend this report to be Rethink Priorities’ final word on malaria-related data systems. We have tried to flag major sources of uncertainty in the report and are open to revising our views based on new information or further research.

Executive summary

Notes on the focus and scope of this report

This project is a landscape analysis of the main challenges and efforts to improve routine data collection systems for estimating the malaria burden in low-income countries. Its two main goals are to:

1. Provide an overview of the main challenges and constraints for collecting health data routinely.
2. Provide an overview of previous or current efforts or initiatives to improve the accuracy of disease burden data.

The focus of this project was guided by what we consider the most decision-relevant aspects for GiveWell: 

• We focus predominantly on malaria, rather than general disease burden, as this is a key funding area for GiveWell and allows the project to be more focused and actionable.

• We focus on the data collection stage in the disease burden estimation pipeline, as our prior is that this stage has the largest scope for improvement.
• Our core focus is annual and national disease burden estimates, rather than higher frequency or more disaggregated estimates.    
• Geographically, we focus our literature review predominantly on sub-Saharan Africa (SSA), as it is home to the vast majority of the malaria burden.
• We focus a larger share of our time on reviewing the challenges and solutions with respect to malaria morbidity rather than mortality, as our prior is that malaria mortality estimates are more difficult to improve compared to incidence or prevalence estimates. 

Key takeaways

• The Institute for Health Metrics and Evaluation (IHME) and the World Health Organization (WHO) both use two malaria burden estimation approaches: Estimates for low-transmission countries are based on (adjusted) national routine data, whereas estimates for high-transmission countries are based on geospatial models by the Malaria Atlas Project (MAP) that rely predominantly on household survey data. Even though the geospatial modeling approach is currently relied on for most malaria cases/deaths, we have seen it heavily criticized. There is a growing reliance on routine data as national surveillance systems improve, though we are unsure about the pace at which this is happening. [more]

• Challenges related to malaria morbidity: Routine surveillance systems in high-transmission countries have challenges at all stages of the malaria routine data flow in terms of coverage, completeness, and accuracy. While some countries’ surveillance systems have shown recent improvements, we are not sure whether and how similar improvements can be made in other countries. Some experts seem rather pessimistic. Household surveys and censuses are also used for modeling, but they have temporal and content-related gaps, which may be possible to resolve with additional funding. [more]
• Challenges to malaria mortality: Malaria mortality estimates in high-transmission countries are based on cause of death fractions, informed by clinical records in vital registration systems and verbal autopsy studies. Clinical records are sporadic, fragmented, and prone to misclassification errors. Verbal autopsy is a rough interim method until vital registration methods are more reliable, but it is very imprecise at determining deaths due to malaria. Minimally invasive autopsy is a promising, emerging method in situations where full autopsy is not possible, but we are unsure whether and to what extent it can be introduced at scale. Case fatality rates are used by the WHO for mortality estimates in low-transmission countries, but they are based on relatively few, old studies, from few countries, and seem fairly noisy. [more]

• A systematic review found that targeted interventions, such as training, data quality checks, and electronic health systems, can improve the accuracy and completeness of routine health data, though evidence is limited and drawn from a small number of studies. [more]
• Multiple large-scale programs aim to strengthen routine and malaria-specific health data systems, but many are heavily reliant on the United States Agency for International Development (USAID) and thus face potential disruption due to recent funding cuts. [more]
• We highlight two case studies of interventions to improve malaria-related data accuracy, based on limited available quantitative evidence. These include a data audit program in Zambia and a system integration project in Burkina Faso. While not necessarily the most impactful or representative examples, they are among the few with measurable outcomes. Additional initiatives are listed in our broader list of potential interventions.
  • Zambia data quality audits (DQAs): Between 2015-2021, Zambia conducted audits comparing paper data on health facility registers to digitized data in the health management information system (HMIS). Repeated audits were associated with better accuracy, but we have concerns about the analysis and do not find it to be particularly convincing. DQAs are likely transferable and easy to implement, and are supported by both the President’s Malaria Initiative (PMI) and WHO, among others. [more]
    • We wonder whether technology might be another way to achieve better accuracy for digitized data, but do not have enough information to be confident about an existing app. [more]
  • Burkina Faso Improving Malaria Care project: Between 2013-2020, Jhpiego led a USAID-funded project to integrate malaria data collection into health management information systems (HMIS), train healthcare staff, and set up data validation procedures. Accuracy of data collection is clearly shown to improve between 2014-2017, and may have reached the project’s target of 90% accuracy by 2020. It’s unclear how transferable this project would be without political buy-in and sustained investment. [more]

Challenges and constraints for malaria-related data systems

In this section, we first outline the main data inputs and processes that are used for both the Institute for Health Metrics and Evaluation (IHME) and the World Health Organization (WHO) malaria burden estimates and explain how these relate to GiveWell’s cost-effectiveness models. We then describe the main challenges and constraints for malaria-related data systems and provide some very rough, low-confidence thoughts on whether and how we expect these could be improved.

Estimates for high-transmission countries rely on epidemiological models based on household surveys, rather than national routine surveillance

[Confidence: High. While we have not investigated IHME, WHO, and MAP estimation processes in detail, we are confident that our summary of their model approaches and inputs is broadly accurate.]

Summary:

Malaria burden estimates by IHME and WHO rely broadly on the same data inputs, and their modeling approaches can be divided into two categories: Estimates for low-transmission countries are generally based on (adjusted) national routine data, whereas estimates for high-transmission countries are based on geospatial models by the Malaria Atlas Project (MAP) that rely predominantly on household survey data. All of GiveWell’s supported country-level malaria interventions are in the latter category. Even though the geospatial modeling approach is currently relied on for most malaria deaths, we have seen it heavily criticized. There is a growing reliance on routine data as national surveillance systems improve, though we do not know the pace at which this is happening. 

The estimation of global and national malaria disease burden figures is a complex process that differs across countries and relies on a large number of different data sources. We spent ~1.5 working days trying to understand what data inputs the IHME and WHO malaria burden estimates are based on. We focused on understanding the data inputs in broad strokes rather than trying to gain a deep, exact understanding of all of the data sources and estimation processes used. Our overall impression is that, while WHO and IHME estimates differ slightly in their precise estimation methods and details, they broadly rely on the same data inputs for generating malaria morbidity and mortality estimates. Thus, we think that the barriers and challenges we outline in the next section hold for both WHO and IHME malaria burden estimates.

Both the IHME and WHO use 2-3 different tiers of malaria burden estimation methods that depend on the data availability and quality across countries (see Appendix A for map overviews of these methodological tiers used across organizations). In Appendix D, we created a rough overview of the data inputs used by IHME and WHO for malaria morbidity and mortality estimates in high- and low-transmission countries, respectively. 

Essentially, the higher the quality of national routine surveillance systems data, the more these burden estimates are based on national routine data. The lower the quality of routine data, the more the estimates are based on household survey data regarding predictors of malaria incidence/ prevalence and causes of death. 

More precisely:

• In very few countries, the national routine data is used as is, without any adjustments (e.g., South Africa). In some countries, the routine data is adjusted for various factors (e.g., treatment-seeking rates), which are based on household surveys, to obtain incidence/prevalence estimates (e.g., Ethiopia, Senegal). Mortality estimates are based on these incidence estimates in combination with estimated case fatality rates (for WHO) and several predictor variables (e.g., proportion of effectively treated fevers, for IHME). These are all low-transmission countries.
• In the vast majority of countries in sub-Saharan Africa (SSA)—high-transmission countries such as Nigeria, Kenya, and Ghana—the quality of surveillance data is considered too low for a reliable estimation of the malaria burden, and morbidity and mortality estimates rely on various other data sources, such as household surveys. This approach accounts for ~90% of estimated malaria deaths (Noor, 2018, p. 9). All of the country-level malaria interventions supported by GiveWell as of 2023 are in this category.
  • Malaria case estimates are derived from parasite prevalence estimates, generated by MAP, which are obtained from household surveys. These parasite prevalence estimates are combined with environmental and sociodemographic covariates to estimate incidence using geospatial modeling techniques.
  • Malaria mortality estimates are predominantly based on a combination of all-cause mortality estimates and estimated fractions of death due to malaria from household surveys.

Thus, malaria burden estimates based on household surveys of predictor variables and epidemiological modeling are the most directly relevant to GiveWell’s cost-effectiveness models, as routine data is currently used only to a small extent for malaria burden estimates of high-transmission countries. However, as countries are improving their national surveillance systems over time, they are expected to transition in the long term from household survey-based parasite-rate estimation methods to methods reliant on routine data or a combination of both. We learned in a conversation with Tasmin Symons (Head of Model Development at MAP) that MAP is doing an increasing amount of modeling with routine data from multiple countries in SSA.

While geospatial modeling based on parasite prevalence rates, such as that done by MAP,

is a widely used approach for estimating malaria burden in high-transmission settings, it has faced criticism from some experts. Bob Snow (Professor of Malaria Epidemiology at the University of Oxford), who founded MAP in 2005 and has decades of experience in malaria epidemiology, has raised concerns that models developed by MAP and IHME rely heavily on complex methods fitted to sparse, outdated, and imperfect data, especially for mortality estimates. According to Snow, these models may not sufficiently incorporate biological or clinical understanding of the disease and are often not thoroughly validated against empirical data. He has also questioned whether the broader modeling community has sufficiently updated its data sources and methods to reflect the evolving health landscape in Africa.

Tom Churcher (Professor of Infectious Disease Dynamics at Imperial College London) has also expressed skepticism about the reliability of geospatial malaria models in high-transmission settings, noting that while the outputs may be statistically robust and as good as we can currently do, the lack of data behind these models means that predictions might be quite different from the on-the-ground reality, particularly at the local level.

When we raised the critique that MAP relies on outdated data, Tasmin Symons explained that this does not accurately reflect their current approach. She clarified that MAP’s models are primarily based on recent cross-sectional data from sources like the Demographic and Health Surveys (DHS), which include parasite rates, care-seeking behavior, and intervention coverage. These are supplemented by ongoing literature reviews and older data sources, such as control arms from randomized controlled trials (RCTs), mainly for retrospective estimates. The use of older data is intentional when modeling past years, like 2000. Symons also emphasized that MAP’s database is continually expanding and currently includes over 56,000 geolocated data points. While some historical datasets still appear in malaria modeling, she noted this is often due to ethical constraints on collecting comparable data today.

The choice of the estimation approach makes a large difference to the burden estimates in high-transmission countries. There is a substantial divergence between malaria cases estimated by adjusted routine data models and by parasite-to-incidence models (see Figure 1 below), as pointed out by Noor (2018, p. 3) in a 2018 talk at a WHO meeting on malaria burden estimation methods using data from 2016. For example, the parasite-to-incidence model estimated around 8,000 malaria cases in Uganda in 2016, while the adjusted routine data model estimated about four times as many cases. We have not seen any more recent comparisons, but would be surprised if the gap has been closed.

Figure 1: Comparisons of parasite rate-to-incidence vs. adjusted routine data model estimates – 2016

Note. From Noor (2018, p. 3)

Challenges related to malaria morbidity

[Confidence: We are highly confident that we identified the key challenges for both routine data and household surveys/censuses, but we are unsure about which challenges contribute most to the uncertainty in burden estimates.]

Summary:

National routine surveillance systems in high-transmission countries have challenges at all stages of the malaria routine data flow in terms of coverage, completeness, and accuracy. On the one hand, several countries’ surveillance systems have shown substantial improvements in recent years, and the WHO Global Technical Strategy (GTS) and the Global Fund to Fight AIDS, Tuberculosis and Malaria consider improving surveillance a priority. However, we are not sure whether and how similar improvements can be made in other countries, and whether and to what extent the WHO GTS and the Global Fund have resulted in actual improvements yet. Moreover, some experts seem rather pessimistic. Household surveys and censuses are heavily relied on for various aspects of the modeling, but they also have several limitations, such as: some key modeling inputs are currently not measured in surveys, and the surveys are run in varying frequencies, which leads to temporal gaps in the data. Two experts we spoke with recommended funding the DHS program as a way to improve data.

In the following, we summarize challenges related to national routine surveillance systems and household surveys and censuses, as our impression is that these are currently the most important data sources used for malaria burden estimation. Other data sources are also used (e.g., environmental data from satellites for geospatial modeling at MAP), though our impression is that these play a somewhat secondary role in the modeling process. We did not review these additional data sources in detail.

National routine surveillance systems are unreliable in most high-transmission countries, and we are unsure to what extent they can realistically be improved in the coming years

To understand the challenges and constraints regarding routine data systems, it is helpful to first understand how these systems should work in the optimal scenario. Alegana et al. (2020) provide such an overview (see Appendix E for a flowchart overview of the “ideal malaria routine data flow” from p. 6 of their paper). According to the authors:  

The use of routine data for malaria morbidity estimation requires an understanding of the denominator population from which the cases originate, completeness and demographics of the number of reported malaria cases, and the uncertainties or biases associated with these quantities. Ideally, all fevers that could be malaria occurring within a community must reach a facility where parasitological testing is provided, and all these events are accurately recorded and stored within a real-time electronic data capture system, such as DHIS2. This is rarely the case in Africa settings, and until this ideal is reached, there is a need to estimate the numbers of fevers not reaching diagnostic centres, the fraction tested, and of those who do not reach testing centres or those untested, the presumed fraction positive (pp. 5-6).

We summarize the various factors that explain why routine systems currently deviate from the ideal malaria routine data flow in Appendix F. Essentially, challenges exist at all stages of the malaria routine data flow, from estimating the size of the denominator population and the number and location of existing healthcare providers via population censuses, to understanding treatment-seeking behavior and malaria testing of febrile populations, and to data reporting of healthcare providers via the DHIS2 platform, an open source health information system. To summarize, the main deficiencies of routine data systems we encountered are related to coverage, completeness, and accuracy (adapted from Alegana et al., 2020):

• Limited care-seeking of patients with fever (~67% of children under 5) makes it difficult to identify all malaria cases. There are ongoing initiatives of community health workers trying to capture malaria cases of people who do not seek care, but we are not sure how well-functioning and promising these initiatives are.

• Limited malaria testing of those who seek care (~57% of children under 5 who seek care), which can be a result of various factors, such as inadequate training and supervision of health workers, and shortages and stock-outs of equipment and malaria tests. Identifying malaria without testing is difficult, as most malaria symptoms are nonspecific and hard to distinguish from many other febrile illnesses. Moreover, not all cases of malaria are symptomatic. According to Gotskind (2019), “in high-transmission areas, asymptomatic malaria is more prevalent than symptomatic malaria,” but asymptomatic malaria can still contribute to transmission. This can, for example, be a result of acquired immunity after repeated parasite exposure.

• Issues with quality, consistency, and completeness of data reported in DHIS2, even though the DHIS2 has been adopted rapidly across most countries in SSA.

Moreover, even if routine data is available, it is difficult to interpret and compare across countries and time, due to country-specific differences in health systems across SSA.

Investing to improve routine data collection appears to be a priority for the WHO and the Global Fund. The WHO GTS (2015) for Malaria 2016-2030 has three pillars, including to “transform malaria surveillance into a core intervention”, which explicitly mentions the need for accurate information (p. 20). Meanwhile, the Global Fund’s  (2021) strategy for 2023-2028 highlights an overall need to “strengthen generation and use of quality, timely, transparent and disaggregated digital and secure data at all levels” (p. 34). Specifically for malaria, the strategy mentions a need to strengthen surveillance, invest in health information systems, and improve data quality and timeliness (p. 28). 

We did not have time to investigate the extent to which identifying routine data collection as a strategic priority has translated into increased investments. According to Alegana et al. (2020, p. 9), even though “surveillance is considered as a core intervention and a third pillar for the GTS [. . .] there continues to remain a focus on what commodities (including their costs) are required for disease treatment and prevention, and less on how to improve disease burden estimation at national levels.” According to the Information Note for Resilient and Sustainable Systems for Health (Global Fund, 2023), countries should allocate approximately 2% of their grant budget to national routine surveillance systems covering all diseases during the 2023–2025 grant cycle (pp. 75–79). The document also recommends additional investments in mortality reporting and data quality improvements. Notably, the Global Fund designates certain surveillance and reporting elements as mandatory requirements for each grant cycle.

Our impression is that experts differ in how optimistic they are about the potential to improve routine health data systems. On the one hand, Bob Snow emphasized that strengthening routine data systems should primarily be the responsibility of national governments, rather than being externally driven by donors, and that such improvements require long-term systemic change tied to broader health system development. He was doubtful that major funders, such as the Gates Foundation, would prioritize this kind of investment. Snow also criticized some donor practices—particularly those that establish parallel data systems managed by external agencies—as counterproductive to genuine system strengthening. On the other hand, Tasmin Symons pointed out that investment in health systems often results in rapid improvements in the quality of routine case data, especially when these improvements are augmented by household surveys. Moreover, Symons mentioned that MAP is doing an increasing amount of modeling based on routine data in SSA. 

In summary, while some experts remain pessimistic about the potential to strengthen routine health data systems, there are pockets of hope, with notable improvements in some countries and increasing use of routine data for decision-making. However, it remains uncertain how replicable these successes are in other contexts. And while the WHO’s GTS and the Global Fund prioritize surveillance, it is still unclear to what extent this has translated into substantial investments or meaningful improvements.

Household surveys and censuses provide various key inputs for burden estimates, but they have temporal and content-related gaps   

Household surveys are used in several ways to estimate the malaria burden. In countries where estimates are based on adjusted routine data, adjustments are typically made with parameters based on household surveys. In countries where estimates are based on a parasite rate-to-incidence conversion, parasite rates and socioeconomic/demographic covariates are also based on household surveys.

While a variety of different household surveys are used by IHME and WHO, our understanding from a conversation with Tom Churcher is that the—by far—main data source is surveys from the USAID Demographic and Health Survey (DHS) program.” The program ran both more general household surveys (DHS), as well as the Malaria Indicator Survey (MIS), which, as we understand, both collected information related to malaria (e.g., parasite prevalence based on rapid diagnostic tests and/or microscopy in the lab and treatment-seeking behavior of children). Our understanding is that the Multiple Cluster Indicator Survey (MICS), run by UNICEF, is also a commonly used survey in this context, though we’ve seen it being mentioned less often. 

Our impression is that there are several key issues with household surveys:

• Some adjustment factors used for routine data are not empirically measured/estimated based on household surveys, but rely on assumptions. For example, the DHS only includes treatment-seeking behavior of children under 5. Treatment-seeking behavior above 5 is not measured. Thus, malaria incidence/mortality for older populations is extrapolated from findings on small children.
• DHS surveys are run in somewhat irregular intervals and varying frequencies, which vary across countries (see here for the numbers and years of surveys across countries). Moreover, as surveys inherently only capture one point in time (and are, in our experience, usually not run several times a year), they don’t capture any seasonality in malaria transmission.
• Parasite prevalence data, which is a major input in MAP’s malaria incidence estimates, has “considerable temporal gaps” (Noor, 2018, p. 10). Parasite prevalence surveys are conducted every 3-5 years, according to Alegana et al. (2020, p. 9). In 2018, malaria incidence estimates were “based on 37 217 survey clusters from 48 countries conducted between 1975 and 2017” (WHO, 2018, p. 5). We provide an overview of the years and countries in which parasite rates were measured in DHS, MIS, and MICS surveys since 2006 in Appendix H. Some countries (e.g., Gabon, Niger) have indeed very large temporal gaps.

Tom Churcher and Tasmin Symons pointed out to us that they recommend supporting/funding the DHS program for more surveys. Funding a DHS survey is estimated to cost about $1.3 million (Chandy and Zhang, 2015, p. 7). We did not investigate this option in detail and are not sure to what extent running an additional survey could reduce uncertainty in estimates.

Moreover, as explained in Appendix F, to estimate the number of malaria cases and deaths, it is necessary to know the size of the population, which is typically done via population censuses every 10 years. In some countries, the size of the denominator population is not well known as censuses are conducted too infrequently (less than once every 20 years). Funding an additional census is estimated to cost about $2 per inhabitant (Chandy and Zhang, 2015, p. 3). However, this seems to be, at least intuitively, a bit too far removed from directly improving malaria burden estimates, and we also expect that the lack of census data might be more a result of political instability in some countries rather than funding constraints. 

Challenges related to malaria mortality

[Confidence: Medium. The literature seems fairly unequivocal on the challenges and possibilities with respect to malaria mortality estimation, but we have not confirmed our impressions with an expert.]

Summary:

Malaria mortality estimates in high-transmission countries are based on cause-of-death fractions. These are estimated based on two data sources: clinical records in vital registration systems (i.e., medical professionals issuing certifications on the cause of death based on, e.g., examinations) and verbal autopsy (i.e., family members are asked about the deceased’s symptoms prior to death in household surveys). Clinical records are sporadic, fragmented, and often prone to misclassification errors. Verbal autopsy is a rough interim method until vital registration methods are more reliable, but it is heavily criticized for being imprecise for determining deaths due to malaria. Minimally invasive autopsy is a promising, emerging method in situations where full autopsy is not possible, but we are unsure whether and to what extent it can be introduced at scale. Case fatality rates are used by the WHO for mortality estimates in low-transmission countries, but they are based on relatively few, old studies, from a handful of countries, and seem fairly noisy.

The malaria cause of death fraction is mainly determined by vital registration systems, verbal autopsy, and minimally invasive autopsy 

Reliable vital registration systems are largely unavailable in SSA, and are improving very slowly

In SSA, vital registration systems that document both mortality counts and causes of death lag behind global standards (Sankoh et al., 2020, p. e33). This holds both for the overall number of deaths (from any cause) and deaths with an identified cause, which are both used for malaria mortality estimates.

The WHO provides guidelines and tools for recording the cause of death. Our general understanding is that a medical certification of the cause of death needs to be issued by a medical practitioner. This is essentially a form that needs to be filled based on, e.g., an examination of the body, testing, and the deceased person’s medical history and clinical records (WHO, n.d.). “Autopsy remains the gold standard for confirming the cause of death” (Cox et al., 2012, p. 1), though our understanding is that autopsies are performed only in special cases, even in high-income countries. While forms can help medical practitioners to determine the cause of death in SSA, “obtaining information on the cause of death remains elusive, on account of deaths occurring at home in the absence of medical attention in remote areas” (Rao et al., 2006).

We have not found any comprehensive overview of the state of vital registration systems in SSA, but various figures point to these systems being highly incomplete and imprecise: 

• According to a 2016 survey by the Economic Commission for Africa (ECA), “only one in three deaths in the region is captured by official registration systems, and [. . .] only 18 of 54 countries record and report annual deaths. Only four African countries have a level of death registration coverage and cause of death information that meets international standards” (Sankoh et al., 2020, p. e33; ECA, 2017). Moreover, “although most African countries have laws dictating that deaths should be registered, few put these laws into use. Only eight of the 39 countries in the ECA study allot adequate financing to civil registration systems, and five have no recurrent budget for these systems. Only 11 countries have electronic recording systems at the local level, and only seven have electronic systems that enable healthcare facilities to report deaths to local registration offices. Moreover, even among the few countries that systematically record deaths, many lack mechanisms to ensure that the data reaches health policymakers” (Sankoh et al., 2020, p. 33). According to a WHO (2020) assessment, there do not seem to have been major improvements since the ECA survey, as “about two thirds of countries in the African region do not have reliable data on births, deaths, and causes of death.”
• According to Suwalowska et al. (2023, p. 3), “causes of deaths are not collected, as many people die in their homes without having been seen by a qualified medical professional and are buried or cremated with their medical history unknown. A study by Adair (2021) has estimated that 60% of deaths in the Global South occur at home, compared with 27% in high-income countries. [. . .] Adding to this complexity is the fact that even for those who die in hospitals, clinical autopsy – a gold standard for determining the cause of death – is rarely conducted due to low acceptability and lack of facilities and trained pathologists. Moreover, diagnostic tools before a death occurs are limited. As a result, cause-of-death assignments on death certification made by physicians are often inaccurate and unreliable.”

According to Sankoh et al. (2020), there seems to be insufficient political will and donor interest to improve these systems. Murray et al. (2014, p. 2) called the progress of the development of vital registration systems over the last four decades “remarkably slow.” Improvements in terms of birth registration coverage in recent years (covering 40% of births in 2012 to 57% in 2015) show, according to Sankoh et al. (2020, p. e33) that “rapid progress is possible—that coverage should now be extended to deaths”. 

Verbal autopsy is a commonly used tool, but inherently imprecise

The verbal autopsy (VA) method was developed as an “interim method” to determine causes of death at the population level until functioning vital registration systems are introduced (Herrera et al., 2017, p. 2). It “consists of an interview conducted with a family member or an individual familiar with the deceased using a structured questionnaire to gather information about the signs and symptoms, and their duration experienced by the deceased, and events leading up to the death. The information collected is used to determine the individual COD [cause of death] using the International Classification of Diseases, Tenth Edition (ICD-10). The COD is assigned either directly by a trained physician or other automated methods” (ibid, p. 2). 

Our understanding is that, in contrast to vital registration systems, VA is usually done via household surveys with family members, rather than with medical professionals, and is not necessarily done immediately after a death, but at whatever frequency a survey is being carried out (which we expect might, in some cases, require long recall times for family members). While VA contains questions about symptoms, it does not, to our knowledge, incorporate any clinical data on the deceased person that could help determine the cause of death. We are not sure whether, in practice, vital registration systems and VA are mutually exclusive concepts or whether combinations of both approaches are used.

VA is a widespread method, particularly in SSA, but it is generally recognized “by the global malaria community that VA does not perform particularly well, regardless of the COD [cause of death] assignment methods used, for determining malaria mortality” (ibid, p. 5). 

A systematic review of 88 studies (adapted from Herrera et al., 2017, pp. 4-8) summarized its many limitations:

• Low and varying levels of sensitivity and specificity for malaria:
  • This is mainly because malaria symptoms are nonspecific and overlap with other causes of death (e.g., meningitis, acute respiratory infections, and HIV), which can lead to misclassification. The misclassification also depends on the epidemiological context, e.g., the seasonality of diseases and whether malaria mortality is identified in a high/medium/low transmission area. 
  • It is also difficult to determine whether malaria was a direct or indirect cause of death in situations with other contributing factors. For example, it is difficult to distinguish between malaria and anemia as direct or indirect causes of death, in cases when a person was likely afflicted with both diseases.
  • VA studies often do not have access to existing medical information, e.g., if an individual received a malaria diagnosis through testing.
• Lack of comparability of findings across sites and studies:
  • There is no commonly agreed-upon standard for using VA methods. VA studies differ in many ways (e.g., targeted age groups, questionnaires used, training provided to physicians, sample selection procedures, length of recall period). Many studies lack transparency about their exact VA approach used and how malaria was determined as a cause of death.
• Inadequacy of clinical records as a comparison group for VA study results:
  • VA validation study results are typically compared against clinical records, which by themselves are prone to misclassification errors. 
  • Moreover, clinical records are often based on different populations than those of VA studies, so their comparability might be limited. Thus, the true misclassification error of VA might be even larger than studies show.  
• General limitations of VA studies:
  • This includes small sample sizes, recall bias, and the fact that many deaths in VA studies remain undetermined due to incomplete information.

Despite these many limitations, some call VA “the best tool we currently have” (Snow, 2014, p. 2; Byass et al., 2013). Our intuitive guess is that some of its limitations can likely be at least reduced. For example, a standardized process for VA methods could be introduced, which would increase comparability across studies and might help reduce misclassification bias (e.g., by allowing better accounting for seasonality effects). Sample sizes could be increased and recall bias reduced (e.g., by using shorter recall periods), but we are unsure whether it would be worth the additional cost. Nonetheless, we expect that issues such as the inherent nonspecificity of malaria symptoms would be extremely difficult to rectify, and thus, VA would remain highly limited as a tool to estimate the burden of malaria.

Minimally invasive autopsy is considered a promising emerging approach, but we are unsure whether it can be introduced at scale

Minimally invasive autopsy (MIA) (also called ‘minimally invasive tissue sampling’ [MITS]) has recently emerged as a “potential new method for determining COD in developing countries where full autopsies are not possible. This technique offers the potential for improved diagnostic accuracy of COD, and could potentially in the long term obviate the need for VA [verbal autopsy] studies” (Herrera et al., 2017, p. 6). It is “designed to determine the CoD mainly in low-resource settings, as a feasible alternative to the complete diagnostic autopsy (CDA). The procedure leaves hardly any visible trace on the body, and thus, is more acceptable than the CDA” (Rakislova et al., 2021, p. 2). 

We found several studies that pointed to MIA as a promising approach that deserves consideration in the context of malaria mortality. For example, Rakislova et al. (2021) tested the accuracy of MIA in determining malaria mortality in Mozambique against complete autopsy. They found a sensitivity and specificity of both 100%, but this is based on only six malaria deaths. The authors also pointed out that “the MIA procedure has been validated in perinatal, paediatric, maternal and other adult deaths in Mozambique and Brazil, showing a moderate to substantial concordance with the CDA, particularly for infectious diseases even when evaluated blindly to any additional data, with such concordance increasing to almost perfect when the clinical information is added” (p. 2), which we did not have time to review.

We have come across several pilot studies investigating MIA in different contexts (e.g., in South Africa [Chawana et al., 2019]), but we are unaware of any concrete efforts to introduce the method at scale and do not have a sense of how promising or realistic this is. Moreover, we are unsure whether this could be used only in clinical settings or also at people’s homes in remote, underserved areas. However, we know that the WHO Global Malaria Programme is aware of MIA and is planning to explore this topic further in the future.

Case fatality rates are based on a small number of old studies with wide uncertainty bands

Both the WHO and IHME rely—at least for some countries—on estimated case fatality rates (CFR) that are combined with incidence estimates to estimate malaria mortality. According to a 2018 report of the WHO Evidence Review Group on malaria burden estimation methods, CFR estimates for malaria are based on a relatively small set of studies and “computed using data from old studies. This approach fails to account for the substantial changes in malaria case management and the obvious variability between countries” (WHO, 2018, p. 1). According to the authors, this “could mean that deaths are being overestimated in some transmission settings” (p. 4).

Indeed, a presentation by Noor (2018), which was part of this 2018 WHO meeting shows that the most recent CFR estimates used for WHO’s 2016 malaria estimates were from the early 2000s and from studies undertaken in a relatively small number of countries (see Appendix B). Moreover, there is a large variation in some of these estimates. For example, CFRs estimated in Kenyan hospital settings range from 3.5% to 33% in the same time period. We have not found more recent information on the reliability and challenges with respect to malaria case fatality rates, but have deprioritized further desk research on this as CFRs only enter malaria burden models for low-transmission countries (which are not supported by GiveWell as of 2023) in WHO models and seem to be an output rather than an input in IHME models.

We have not found any overview of the specific challenges related to estimating case fatality rates, but we expect that case fatality rates suffer from many challenges previously described in the section on challenges with routine surveillance (e.g., limited use of the formal healthcare system by many patients, limited testing of patients). 

Efforts and initiatives to improve malaria-related data systems

To explore efforts to improve malaria-related data systems, we began with a literature search focused on data collection, malaria, and disease burden. To understand general strategies for improving routine health data quality, we specifically looked for systematic reviews. The most relevant we found was Lee et al. (2021), which is summarized in the next section. We then identified major initiatives focused on strengthening health data systems, especially those targeting malaria. We found these through internal team knowledge and approximately 3.5 hours of desk research. The initiatives are discussed here.

Routine health data quality can improve through targeted interventions addressing accuracy and completeness, but the evidence is limited

Lee et al. (2021) conducted a systematic review on interventions aimed at improving Routine Health Information Systems (RHIS). They identified 56 studies focused on increasing data quality—44 of which included quantitative outcome measures—and grouped them into 17 distinct intervention components (p. 4). These components are quite broad, including activities such as “meetings” (e.g., data discussion meetings, dissemination meetings, or quarterly reviews), “providing training,” and “using an electronic health information system” (eHMIS) (p. 1). We provide a full list of the components and more detail on the definitions and results in Appendix C.

The authors classified outcomes into three dimensions of data quality: accuracy, timeliness, and completeness. Of the 44 studies, 93% reported some improvement in data quality. To assess which interventions were most effective, the authors focused on studies where the outcome exceeded a threshold of 80%. However, it is not clear how this 80% threshold was defined or calculated. They then examined how frequently each intervention component appeared in the “above threshold” studies versus those that did not meet the threshold (p. 9).

The intervention components most strongly associated with improved accuracy were:

• Training
• Equipment purchase and maintenance
• Data quality checking

Those most associated with completeness were:

• Meetings
• Stakeholder engagement
• Use of an electronic health management information system (eHMIS)

In general, having more than four intervention components was positively associated with improved data quality. However, some components—such as data quality assessments and database harmonization—were linked to improvements in fewer than half of the studies where they were applied (pp. 8–9).

While this evidence is based on 44 different studies, we do not have high confidence in the robustness of its findings. Most intervention components were implemented in only three or fewer studies, limiting the strength of the conclusions.

Lee et al. (2021) cover general health data systems. We also reviewed the included studies to identify those specifically focused on malaria data. Six studies were malaria-related, of which three may be useful as case studies on improving malaria data:

• De la Torres et al. (2014): Examines the use of a mobile reporting system in Mali, as part of the MEASURE Evaluation initiative (described in the next section).
• Chicha et al. (2015): Describes the implementation of a surveillance and feedback loop system in Zambia, supported by the President’s Malaria Initiative (see next section).
• Tobgay et al. (2016): Covers the introduction of mobile and web-based technology for malaria surveillance in Bhutan, with support from WHO and the Asia Pacific Malaria Elimination Network (APMEN), funded by the Australian government (p. 7).

Routine and malaria health data systems are the focus of numerous programs—now facing potential disruptions from USAID cuts

We identified a number of organizations and initiatives aimed at improving routine health data, including six that focus specifically on malaria. Nearly all of these initiatives list USAID or the Gates Foundation as major funders, and we also have reason to believe that the Global Fund encourages countries to allocate grant funding toward strengthening routine health data. Below, we list some of the larger initiatives we identified (with more detail available in this spreadsheet) and provide additional examples of smaller or more targeted efforts in Appendix G.

Initiatives focused on improving malaria data

The Malaria Atlas Project (MAP) is an international research collaboration focused on malaria data. It partners with the WHO and IHME (which runs the Global Burden of Disease [GBD] study) to create annual burden of disease estimates for malaria. The Gates Foundation is MAP’s main donor and committed approximately $8.2 million in 2018 for a 56-month period. 

The President’s Malaria Initiative (PMI) primarily focuses on prevention and treatment, but its efforts also include activities to improve data accuracy. PMI’s budget was about $746 million in fiscal year 2022, though we believe only a small share was allocated to improving data (PMI, 2022, p. 4). Bob Snow has expressed criticism of PMI’s approach to routine data improvement, arguing that their model relies heavily on externally funded agencies to collect data and report it back to the US, rather than supporting long-term, locally led system development. He views this as an ineffective strategy for building sustainable data capacity.

PMI’s Measure Malaria program specializes in malaria surveillance, monitoring, and evaluation. It embeds senior advisors within national malaria control programs and provides technical support to improve data collection, analysis, and use. The program also supports malaria strategy development and hosts specialized workshops. Measure Malaria is a five-year project (June 2019–June 2024) with a funding ceiling of $35.9 million, currently operating in 10 high-transmission countries in SSA, including the DRC. 

In addition, PMI supports malaria diagnostic testing by offering training, updating national policies, and assisting with procurement and diagnostic equipment (such as microscopes and rapid diagnostic tests).

The Clinton Health Access Initiative (CHAI) received an $8.5 million grant from the Gates Foundation in 2022 for “Strengthening malaria data and digital health systems” in Mozambique, with a 48-month commitment. We have not found more details on this program on the CHAI website. CHAI also piloted a project in Uganda to improve data reporting by private drug shops and clinics (CHAI, 2022). In an interview, Justin Cohen noted that CHAI has supported malaria surveillance systems in more than 10 countries (CHAI, 2023).

The introduction of mobile and web-based technology for malaria surveillance in Bhutan was one of the initiatives described in the previous section. It was funded by the Asia Pacific Malaria Elimination Network (APMEN). 

Finally, the introduction of digital field data collection systems into seasonal malaria chemoprevention campaigns was implemented by the National Malaria Control Programmes in Ghana, Benin, The Gambia, and Nigeria (Balla et al., 2021). 

Initiatives focused on improving general routine health data collection

Several initiatives aim to strengthen general routine health data collection practices, which may include malaria-related indicators. We summarize selected efforts below. We did not explicitly check that each of the initiatives listed includes malaria data.

The GBD study partners with national institutes to improve data collection processes. One example is the “burden of disease unit” at the Ethiopian National Data Management Center for Health (NDMC), which collaborates with the GBD study and is responsible for health data collection and archiving. Our impression is that MAP functions as the malaria-focused team within the broader GBD study. We expect more such collaborations between GBD or MAP and national data institutes to exist, but did not explicitly search for them.

The Demographic and Health Surveys (DHS) program collects, analyzes, and disseminates data on population and health in over 90 countries. It offers technical support for surveys, many of which include malaria data. A figure showing where and how often surveys that collect malaria prevalence data are conducted is available in Appendix H. USAID was the main funder of the DHS program. We searched briefly (~20 minutes) but were unable to find information on their total budget.

DHIS2 is an open-source health information system used in more than 80 countries to collect and analyze health data. It has several funders such as Norad, the University of Oslo, the Research Council of Norway, PEPFAR, the Global Fund, UNICEF, CDC, Gavi, and the Gates Foundation (DHIS2, n.d.). We were unable to find public information on the total program budget, but the Gates Foundation committed approximately $4.3 million across three grants in 2021 and 2022 for a period of about 40 months.

Bloomberg Philanthropies’ Data for Health initiative provides tools and training related to health data. Publicly available information on the program’s structure and outcomes is limited. The Gates Foundation committed $20 million to the initiative in October 2021 for a period of 21 months.

USAID also funded a number of programs aimed at improving health data quality, though we have not been able to identify the budgets for each.

• Data for Impact (D4I) was a six-year USAID-funded program scheduled to end in March 2025. D4I aimed to support countries in generating and using high-quality data to improve health programs and policies and strengthen the technical and organizational capacity of country partners.
• Country Health Information Systems and Data Use (CHISU) aimed to improve the integration and harmonization of systems across health areas to enhance data collection, analysis, and use at national and sub-national levels. At the time of our original assessment in 2023, CHISU operated in 13 countries, including malaria-focused work in high-burden settings such as the DRC, Burkina Faso, and Ghana.
• MEASURE Evaluation was a USAID project that operated from 1997 to 2019, initially focused on family planning but later expanded to broader global health data. It was succeeded by projects such as PMI Measure Malaria, D4I, and the TB Data, Impact Assessment and Communications Hub (TB DIAH).

Multiple opportunities exist to improve malaria data quality along the pipeline

We created a simplified representation (see Figure 2) of the data collection and estimation process for malaria, highlighting key steps and potential opportunities for improvement. The diagram illustrates how estimated incidence and mortality—whether from MAP, IHME, or WHO—rely on parasite prevalence data, covariates, and cause of death information. 

Meanwhile, routine data is collected across various settings (often initially on paper and later digitized, though not always), and then aggregated. As we mentioned previously, Tasmin Symons also described that in some cases, MAP has begun partnering with countries to produce updated estimates based on raw routine data, which can then feed into decision-making.

Figure 2: Simplified representation of the data collection and estimation process for malaria, and potential avenues for improvement

Note. Figure was developed by the authors.

The figure highlights eight points where targeted interventions could improve data quality: 

1. More survey data (specifically on prevalence)
2. Better covariate/other information
3. Improved cause of death data
4. Enhanced modeling for MAP estimates
5. More accurate diagnosis and attribution of cause of death
6. More accurate data entry (especially during digitization) 
7. More complete routine data collection
8. Improved modeling of MAP combined estimates

Ideas and examples for addressing each of these are provided in Appendix G, based on expert interviews, literature review, and our own analysis. 

Two case studies of interventions with evidence of success

To illustrate examples of interventions to improve malaria-related data, we selected two case studies from our list of potential interventions based on whether we found evidence of effectiveness. However, this proved challenging, and in the end, we conducted only two case studies, both focused on “more accurate data entry” (Step 6 in the previous section).

Due to this limitation, and the fact that we found it difficult to gain expert interviews during the project, we have low confidence that these examples represent the most promising or impactful interventions. Nonetheless, we believe they illustrate two distinct approaches that may warrant further exploration: one programmatic (routine data quality audits in Zambia), and one technological (ScanForm for digitizing paper records).

Case 1: Zambia – Data quality audits

Routine data quality audits (rDQAs, or DQAs) are commonly cited as methods to improve malaria data quality and appear to have been implemented in multiple countries by various organizations. At a high level, these audits involve comparing registers at health facilities (where data is initially collected, often on paper) with records in HMIS. Following the audit, actions are taken to correct the data and improve processes going forward. 

Zambia has operated a DQA program since 2015. Our impression, based on a brief from PMI’s Program for the Advancement of Malaria Outcomes (PAMO), is that the program is at least partially funded by PMI, and implemented by PATH in coordination with the Ministry of Health (PAMO, n.d.).

We identified two sources summarizing the program’s activities and results: a brief from PAMO (n.d.), and a presentation from PATH (2023). The program covers 33 districts across four provinces, conducting audits on six months’ worth of data in 10-13 health facilities per district each year. Each facility is visited once every two years. The program has expanded from 155 facilities in its first year to 645 facilities by 2021, auditing facilities up to seven times. The focus is on three malaria indicators: total outpatient attendance, cases tested using rapid diagnostic tests (RDTs), and RDT-positive cases. After each audit, the “DQA team walks the health facility staff through identified errors and provides mentorship to correct them. Together, they make a list of issues to monitor between DQAs” (PAMO, n.d., p. 1). The project accomplished 1,781 DQAs over seven years across 645 health facilities. 

The two sources we found report outcomes using different metrics, as detailed below. Note that in both cases, no formal thresholds for success are provided. 

Results from PATH presentation, 2015-2021 (PATH, 2023):

• Metrics used to measure success: 

• Weighted average percentage error (WAPE) for three malaria data elements and an aggregate, translated into “overall weighted data reporting accuracy”

• Facilities were considered to have:

• High accuracy if WAPE > 85%

• Low accuracy if WAPE < 70%

• Results: 

• Clear difference between results for health facilities at first audit visit (34% low accuracy, 24% medium accuracy, 42% high accuracy) and those at their sixth or seventh visit (low accuracy ≤ 10%, high accuracy ≥ 75%); however, we have concerns about the sample size and selection methodology 

• Comparing 2015 and 2021: 

• Average weighted data reporting accuracy increased from ~70% to ~80%

• Proportion of health facilities with low accuracy dropped from 39% to 20%

• Proportion of high-accuracy facilities increased from 35% to ~60%

• The sample expanded over this time, so these are not direct comparisons

Results from PAMO brief, 2017-2019 (PAMO, n.d.): 

• Metrics used to measure success: 

• Completeness (all elements reported)

• Accuracy (match between registers and HMIS)

• Timeliness (submission by reporting deadline)

• Results: 

• • Timeliness: Broadly unchanged at ~85% facilities on time
  • Completeness: Increased from 89% to 100% of facilities reporting all required elements
  • Accuracy: Monthly accuracy increased from 26% to 54%

While the PATH results suggest that repeat audits at the same facility may increase data accuracy over time, our concerns regarding the analysis limit the confidence we place in these findings. Additionally, even if we had more confidence in the results, we would remain uncertain about the long-term impact of DQAs. We are not aware of any analysis assessing, for example, whether data quality remains improved three years after a single audit.

The underlying approach appears to be both transferable and relatively straightforward to implement. Its usefulness depends on the extent to which a country digitizes a significant amount of data. Our impression is that digitization of routine health data is now very common in SSA, as “DHIS2 is now implemented in all but four African countries” (WHO, 2022, p. 21), though we expect that levels of digitization vary across indicators and locations (we did not investigate this in detail). There are numerous examples of similar activities across countries; for example, the PMI Measure Malaria project has developed an Excel tool for DQAs and rolled it out in six project-supported countries. The WHO (n.d.) also appears to have its own data quality assurance toolkit. However, the PATH (2023, p. 2) analysis of the Zambia program does note that “rDQAs are resource intensive.”

Regarding PMI Measure Malaria’s DQA tool, we could not find an assessment of the impact of implementation on data accuracy: However, their website suggests that in Côte d’Ivoire and Cameroon, data quality was found to be higher in PMI-funded districts using this tool, and these improvements prompted implementation in some regions funded by Global Fund. The team has also developed an Android app for DQAs that was piloted in summer 2022, but it’s not clear that this has been rolled out anywhere (PMI, n.d.)

Interestingly, the Global Fund (2023) Information Note for Resilient and Sustainable Systems for Health indicates that a national Data Quality Review, and development of an improvement plan, is mandatory once per grant cycle (p. 76). It suggests that for this to be high impact, countries should spend $0.5 million (not including implementation of the improvement plan), but it does not include information about the specific activities required.

A potential alternative: ScanForm 

ScanForm could present a technical solution to the issue of accurate data entry from registers. The system involved redesigning paper forms to allow the app to digitize photos of completed (handwritten) documents. The software produces automated data quality reports, covering accuracy, completeness, and timeliness. 

Their website claims impressive character recognition (99.8%) and twofold improvements in accuracy over manual data collection, but no detail on how they came to these conclusions. A ScanForm (2023) presentation gives some examples of character recognition (slides 17-19). A brief search did not identify any independent analysis.

Based on the programs listed on the ScanForm website, the app is predominantly being used in Kenya, but also seven other countries—notably Nigeria, Malawi, and Burkina Faso. It is also being used in clinical trials, e.g., an antimalarial study led by LSHTM and LSTM in Burkina Faso and Kenya. The app was a winner of the MIT Solve Challenge. Its website lists the CDC, PMI, WHO, Global Fund, Gates Foundation, and LSTM as partners, among others. 

We are not confident that this technology has been proven to improve accuracy, but it may be an interesting option to explore with more time—for example, by contacting investigators involved in the clinical trial to gather additional perspectives. 

Case 2: Burkina Faso – “Improving Malaria Care” project

Between 2013 and 2020, Jhpiego led the “Improving Malaria Care” (IMC) project to reduce malaria morbidity and mortality in Burkina Faso. The project was funded by PMI, with a total budget of $19.8 million (PMI, n.d., p. 1), although it is not clear how much of this was allocated to the project’s intermediate goal of improving national capacity for malaria data collection and use.

Based on Jhpiego’s description of the project in Dodo et al. (2021), efforts related to improving data accuracy entailed: 

• Integration of malaria reporting into the national HMIS (hosted by DHIS2), including: 
  • Definition of key indicators for integration
  • Revision of tools and forms
  • Definition of data validation processes
  • Creation of a malaria-specific module within DHIS2
• Revision of the relevant manuals for data entry
• Development and integration of a surveillance, monitoring, and evaluation module into national malaria training
• Training of over 1,800 healthcare workers across 70 districts, including all data managers
• Development and implementation of district-level procedures for malaria data review and validation

According to a progress report (USAID, 2017, pp. 16-18), the metric for accuracy (or precision) is measured “by counting the cases for each indicators (sic) and comparing them with the reported value at each level.” 

The acceptable standard for success is defined as 90% accuracy, and Figure 3 below shows that although results improved after 2014, accuracy had not reached this threshold for any malaria indicator in 2017. We were not able to find a similar set of results for 2020, when the project ended, but Dodo et al. (2021) report that a 2019 evaluation of malaria data quality found “data accuracy above 90% for antenatal care (ANC) services.” It is unclear how comparable these figures are, as we lack information about that evaluation’s methodology (e.g., sample size, indicators measured, and metrics used), but this suggests the project was likely successful. 

Figure 3: Malaria data accuracy for six malaria indicators, in 2014 (blue) and 2017 (red) 

Note. From Brieger (2018). We use this figure rather than the original figure in the 2017 Jhpiego report (p. 17), because Brieger’s version is already in English. The underlying data is the same.

Jhpiego-led, USAID-funded work to improve malaria data quality and use has continued in Burkina Faso under the “Integrated Family Health System” project (2021-2026). We have not investigated this further.

It is unclear which elements of the IMC project were the most impactful. If we consider all of the activities as a package, the solution seems like it could be challenging to implement elsewhere without significant political will and sustained investment involvement over several years. While the general approach might be transferable (i.e., clearly define HMIS processes and conduct training and data validation), we would expect that significant tailoring would be needed in each country.

Appendices

Appendix A. Malaria burden estimation approaches used by IHME and WHO across countries

The IHME uses two categories of estimation methods (Weiss et al., 2019, pp. 324-325; see Figure A1 for a map view):

Category 1 – Surveillance approach:

In the surveillance approach, IHME jointly models national-level incidence and parasite rates (mortality is modeled from incidence) based on reported malaria cases from routine surveillance and peer-reviewed publications. For countries with deficiencies in routine surveillance systems, reported case data are adjusted for various factors (e.g., treatment-seeking rates) in accordance with the WHO adjustment approach. These estimates are then “spatially disaggregated [. . .] to produce high resolution maps.” The surveillance approach is used in countries with “widely available and reliable” routine data (pp. 324-325).

Category 2 – Cartographic approach:

In the cartographic approach, IHME maps parasite rates at the pixel (5 km x 5 km) level and converts these into incidence and mortality estimates. This approach is used in 36 countries in SSA.

Figure A1: Map of IHME categories of malaria burden modeling strategies

Note. From Weiss et al. (2019, Supplementary appendix, p. 72)

The WHO uses three categories of estimation methods (Alegana et al., 2020, p. 5; see Figure A2 for a map view):

Category 1 – Unadjusted routine data: 

• The WHO uses unadjusted routine data on malaria morbidity and mortality from countries with well-functioning surveillance systems and low malaria incidence (South Africa, Swaziland, Comoros, and Djibouti). 

Category 2 – Adjusted routine data:

• The WHO uses national routine data and adjusts it for various factors, e.g., test positivity rate, treatment-seeking rate (Botswana, Eritrea, Ethiopia, The Gambia, Madagascar, Mauritania, Namibia, Rwanda, Senegal, and Zimbabwe).

Category 3 – Estimation using parasite rate-to-incidence:

• The WHO uses “modelled predictions from a composite of interpolated, modelled parasite infection prevalence surveys undertaken infrequently, and transformed to case incidence using a modelled non-linear relationship between parasite prevalence and active case detection from 30 epidemiological studies undertaken between 10 and 20 years ago” (p. 5). This is used for the vast majority of countries (30) in SSA. These estimates are taken from the Malaria Atlas Project (MAP).

Figure A2: Map of WHO categories of malaria burden modeling strategies

Note. From Alegana et al. (2020, p. 5)

Appendix B. Case fatality rate data sources used for WHO malaria estimates

Figure B1: Case fatality rate data sources for 2016 WHO malaria estimates

Note. From Noor (2018, p. 13)

Appendix C. Summary of findings from the Lee et al. (2021) systematic review

Table C1: Summary findings by Lee et al. (2021) per intervention component

Note. The cells show the number of studies with an intervention that did or did not improve data accuracy and completeness above the threshold. For intervention components where data quality improved to above the threshold in less than half the studies, cells are red. When no studies with that data component were included, cells are gray. The definitions of each intervention component can be found in the study’s Supplementary Information (p. 13).

Appendix D. Data inputs for WHO and IHME malaria burden estimates in sub-Saharan Africa

Table D1: Data inputs for WHO and IHME malaria burden estimates in sub-Saharan Africa

Notes: WHO information is adapted from WMR (2022, pp. 132-136) and Noor (2018, pp. 8-15). IHME information is adapted from Weiss et al. (2019, pp. 324-327) and its Supplementary appendix. In cases where routine data is adjusted (for both WHO and IHME estimates), the number of reported malaria cases is adjusted for the “reporting completeness and likelihood that cases are parasite positive” (WMR, 2022, p. 132). This information is reported by national malaria programs. Further adjustments are made for treatment-seeking rates in the public and private sectors based on self-reported fever from household surveys (Noor, 2018, p. 9).

Appendix E. Ideal malaria routine data flow

Figure E1: Ideal malaria routine data flow

Note. From Alegana et al. (2020, p. 6)

Appendix F. More detailed summary of barriers to creating reliable routine surveillance for malaria morbidity burden estimation

In the following, we summarize the factors that explain why routine systems are currently deviating from the ideal malaria routine data flow (adapted from Alegana et al., 2020, pp. 6-8): 

Denominator population:

• The “population denominator from which malaria cases arise” is not always known in some countries. According to the authors, population censuses are conducted too infrequently (i.e., less than every 20 years) in some countries. Moreover, if census data is measured, “fine-scale census data is often not available or accessible to NMPs [national malaria programs]”. 
• The authors point out “new innovative methods of mapping population combining social media platforms with satellite remote sensing via machine learning methods, or triangulating data from human settlements with mobile phones,” which could be explored as a solution (pp. 6-7).

Master health facility list:

• A list of all healthcare providers in a country is necessary to gauge the completeness of routine data reported by healthcare providers. Such master health facility lists are not yet available in many countries, but “censuses of healthcare providers are increasing in scope and coverage across Africa, through the Master Health Facility List (MHFL) initiative. MHFL has been established and updated in 11 countries” as of 2020 (p. 7).

Variation in fever treatment-seeking behavior:

• According to the most recent World Malaria Report, only ~67% of children in SSA with a fever sought treatment in 2015-2021 (WMR, 2022, p. 73). Some people, e.g., those in semi-immune populations, may not seek treatment, as their malaria fever goes away on its own. Others seek treatment, but do so from various different providers (e.g., formal/informal healthcare providers, shops, drug vendors). This data is typically collected in household surveys on actions taken regarding fevers of children in the past 2 weeks, but this information is typically not collected on people older than 5 years. According to Alegana et al. (2020, p. 7), “there is a need to understand treatment choices to define malaria fevers likely to be missed through routine data. This will require more in-depth quantitative survey questions combined with qualitative methods across all age groups.” 

Limited malaria testing for those who seek care:

• Not all fevers reaching healthcare providers are tested for malaria parasites. According to the World Malaria Report, 57% of children under 5 years with fever who sought care received a finger or heel prick for diagnosis from 2015-2021 (WMR, 2022, p. 73). This is despite changes in international malaria case-management guidelines to “improve parasitological testing and treatment adherence to malaria test result” (Alegana et al., 2020, p. 7) since 2011. Variation in testing rates (within and across countries) can be a result of “inadequate training and lack of supervision of healthcare workers, shortages and stock-outs of equipment and mRDTs, and patient-level factors,” and can be mitigated by “improving in-service training, stock management and logistics” (ibid).
• There are ongoing efforts to increase the use of mRDTs through community health workers to capture malaria cases of those who don’t seek treatment in formal care or at all. These tests are reported to the DHIS2 (p. 8), but we are unaware of how much testing is being done by community health workers and how many patients remain untested.
• Malaria burden estimates rely on various parametric assumptions that have not been tested, or at least not sufficiently. For example, a common assumption of WHO malaria burden estimates is that “the fraction of parasite-positive fevers in the formal health sector are the same as those who remain untreated or treated in the informal sector,” but little is known about whether this is actually the case (p. 8). Moreover, models typically assume that parameters (e.g., related to treatment-seeking behavior) are uniform within countries, for a lack of more geographically disaggregated data.

Coverage of routine data for decision-making in DHIS2:

• Although the DHIS2 has been quickly adopted across most SSA countries, various issues remain, such as poor data quality, inconsistencies in the data, and delays in reporting. Incomplete reporting is also “common across all surveillance systems” (p. 8): some facilities do not report any data, some facilities miss certain months or certain data elements. In some countries, multiple data systems and platforms are used in parallel.

Appendix G. Potential interventions to improve data quality

We created a simplified representation of the data collection process and identified steps in the chain where there were opportunities to improve data quality. The diagram and a narrative description can be found here. Below, we list ideas that we gathered throughout the course of our research, based on interviews, the scientific literature, and our own thinking. In many cases, we include minimal additional information, but we felt it could be helpful to capture them at this stage. As we are focused on data accuracy, we have excluded interventions that aim to improve the visualization of data and the use of data in decision-making. We have also deprioritized interventions that improve timeliness of data collection.

• More survey data (specifically prevalence):

• Fund additional DHS/MIS surveys
• Add items to surveys that would be helpful for modeling (described below)
• Use testing methods that can pick up other parasites (e.g., vivax) or mixed infections

• Better covariate/other information:

• Improve access to more granular census and meteorological data (Alegana et al., 2020)
• Better defining population denominators and catchment areas (Alegana et al., 2020)
• Conduct cross-sectional assessments of (especially private sector) drug quality

• Better cause of death data:

• Minimally invasive autopsy 
• Standardized verbal autopsy approaches

• Improved modeling for MAP estimate:

• None specifically identified and out of scope for this report

• More accurate diagnoses/identification of cause of death (for routine data):

• Use testing methods that can pick up other parasites (e.g., vivax) or mixed infections

• Minimally invasive autopsy
• Clear/standardized definition of a malaria case
• Use automated RDT readers (e.g., Deki Reader)

• More accurate data entry:

• Data quality audits (discussed here) 
• Provide training at health centers 
  • USAID-funded Country Health Information Systems and Data Use (CHISU) Program conducted training in six sub-districts in Ghana, and increased cumulative accuracy from 51% to 85% (register versus DHIS2), see here.
• Developing tools to track data quality (Alegana et al., 2020)
• Introduce automatic data entry following testing (e.g., Deki Reader)
  • According to Tasmin Symons, there have been concerns in the past that healthcare workers may ignore negative RDT results, and instead record a positive malaria case and prescribe antimalarials. Automating data entry following testing could lead to more accurate data and limit overprescription. 
  • Adah et al. (2018) investigated this in Nigeria in 2016 and found that the Deki Reader test positivity rate was 23.6%, whereas it was 51.6% in the DHIS (though the DHIS population of patients was larger). We have not reviewed this in detail. Van Dujin et al. (2021) also investigate use of connected diagnostics to achieve similar aims in Kenya. 

• More complete data:

• Encourage those with malaria to visit health centers so they can be recorded
  • In some contexts, offering free treatment for children under five has been an effective strategy to increase the number of people seeking care at public health facilities. However, efforts to redirect people to the public sector should be context-specific, as in countries with a strong preference for private treatment, this approach may not be suitable. A study by Ouédraogo et al. (2020) found that a two-fold increase in confirmed malaria cases was significantly associated with the introduction of such policy changes.
• Enroll a wider range of healthcare providers in the DHIS2
  • Example of proposed intervention in Tanzania, where a private reporting system exists but was not integrated at the time
  • This appears to be a priority for the Global Fund: “While [increasing availability of data] will entail continued investments in routine HMIS, reviews, facility and community surveys and evaluations, greater attention will be given to improving the integration of community data [. . .] and private sector data” (Global Fund, 2023, p. 34). 
• Interpolation of missing data
  • Example of prediction for Burkina Faso to account for missing data due to workers’ strikes (Rouamba et al, 2020)
• Improve completeness of data collected in the public sector
  • Burnett et al. (2019) report that a PMI-funded Malaria Care Electronic Data System was used during supervision visits across seven countries in Africa, which improved completeness from 42% to 89%.
• Reduce the logistics/bureaucratic overheads of reporting to DHIS2 (Tasmin Symons)

• Improved modelling for MAP combined estimate:

• Better understanding of how population being captured in routine surveillance data is/is not representative of the overall population
  • Specifically indicated by Tasmin Symons: Key information needed is what proportion of adults get malaria and seek care at a clinic; also, information about seasonal variation
  • From Alegana et al. (2020, Table 2): “Improved understanding of fever incidence, injection risk and treatment-seeking patterns across all age groups and genders, including better structure quantitative and qualitative methodologies”

Alegana et al. (2020) also mention the following, which do not fit neatly into any of the categories mentioned above: 

• Geo-coding inventories of health providers, and 
• Building capacity in national malaria control programmes

Appendix H. Malaria survey data by country, over time

Based on results from the DHS Survey Characteristics search, we identified the DHS, MIS, and MICS surveys that include information about the malaria parasite rate (whether confirmed by RDT or microscopy). The two charts below show the total number of surveys completed since 2006 (by status), and the timing of surveys in each country over a given period. At the time of writing, 80 DHS/MIS/MICS surveys that collect malaria prevalence data have been completed in 28 countries since 2006. 12 such surveys are ongoing. 

Figure H1: Malaria surveys capturing prevalence since 2006

Note. Total surveys between 2006-2024, based on results from the DHS Survey Characteristics search. Includes DHS, MIS, and MICS surveys that collected malaria prevalence data. 

Table H1: DHS survey completion between 2006 – 2024

Note. Based on results from the DHS Survey Characteristics search. Includes DHS, MIS, and MICS surveys that collected malaria prevalence data. Merged cells reflect surveys titled as running across multiple years. Green indicates the survey is complete, yellow indicates that it is ongoing (final report not yet available), and red indicates that the survey is under audit.