Measuring Counterfactual Impact in Effective Giving: Methods Review and Recommendations

Executive Summary

What We Did

• The problem. Effective giving organizations report impact as “money moved,” but without adjusting for the counterfactual fraction (CF), this overstates the organization’s role in causing donations. How much of observed giving would have occurred anyway, and to what charity? Current practices to answer these questions vary widely and are not directly comparable across organizations.
• Our methods. We collected donation data and CF estimates from 22 effective giving organizations across 13 countries (2022–2026), and recalculated each organization’s estimate on a common methodological basis (standardized referral categories, calibrated CF weights, donation-value weighting) to separate differences in donor populations from measurement artifacts.
• Post-hoc methods. We also developed interim estimation frameworks for three complications that standard checkout surveys do not address: tax deductibility, pledge partnership attribution, and recurring donors.

What We Found

Three questions frame this report: whether CF measurement can be standardized, how much of the variation across organizations is real versus artifact, and how to handle complications that checkout surveys cannot capture.

On standardization

• A standard method is feasible. Data from 22 organizations was enough to build a shared baseline. From this common starting point, an organization can test its own donor patterns against field-wide numbers rather than assumptions.
• The standard assumption that effective altruism (EA) community referrals have zero counterfactual value is empirically contradicted. Combining data from two survey questions asked by one organization (N ~4,000) shows approximately 40% of EA-referred donors would have given to conventional charities or not at all without the organization. Our recommended weight for EA-referred donations is 0.30 (provisional; single-org basis; expert-adjusted for field plausibility).
• Donor-weighted CF overstates donation-weighted CF for 7 of 8 organizations, by 3.6–12.0 percentage points (pp), because large donors disproportionately arrive through lower-CF channels. One organization reverses this pattern because most of their donations come through high-CF channels. Orgs that can only report donor-weighted CF should apply a downward adjustment to money-moved estimates if they know their large donors have lower CFs.
• The impact of uncertainty about non-respondent behavior is negligible for high-response organizations and substantial for low-response organizations. Response rates range from 24% to 82%, reported on inconsistent bases across organizations. Where response rates exceed 70%, even worst-case assumptions about non-respondents shift the estimate by at most 15.5pp; at the lowest observed rates (around 25%), the estimate could sit anywhere in a 39pp range.

On real vs. artifactual variation between organizations

• Around 60% of the apparent spread in CF estimates across organizations is methodological, not based on underlying donor differences, though this figure may be overstated where CF weight variation partly reflects within-category differences across countries. After harmonization, CF estimates among the 8 referral-survey organizations range from 36%–70% (before, 39%–85%) and cluster around a cross-organization mean of 53.7%.
• EA community penetration is the dominant driver of remaining variation. Organizations where EA community referrals account for more than 50% of donation value sit 13–17pp below the mean across organizations. Large divergences from the mean are concentrated in organizations with unusually high or low EA community referral shares.

On post hoc methods

• In seven countries with donation tax relief schemes, CF estimates require a modest downward adjustment. Some giving is tax-motivated; donors may have found another deductible vehicle regardless. Applicable in France, New Zealand, Australia, Germany, the Netherlands, Norway, and Denmark.
• Pledge attribution and recurring donor CF have no empirical grounding in current organizational data. Both are suggested frameworks based on theory that require validation.

Recommended Methodology

• Our recommended approach uses two checkout questions (referral source and counterfactual intent), donation-value weighting, and a 19-category referral taxonomy with calibrated weights. Full implementation guide in Section 7.
• The EA community referral weight has empirical support at 0.30, revised upward from the field’s assumed 0.00.
• Money-moved estimates should use donation-weighted CF, since donor-count weighting overstates donation-weighted CF.

Key Limitations

• The EA community weight (CF = 0.30) rests on a single organization’s data. This is the most consequential parameter in the framework.
• The direction of non-response bias is unidentified for most organizations. Regression-based analyses at two organizations find little evidence that response is correlated with donation size based on observable factors, but unobserved factors could still be motivating a response/donation decision.
• We developed the pledge attribution and recurring donor frameworks based on theory. Neither has been empirically validated.

Priority Future Research

See Section 8.2 for more detail on all priority studies.

• Non-response and hypothetical bias (highest priority). We don’t know whether non-respondents differ in CF from respondents. It is the biggest remaining gap that could realistically be closed. An incentive experiment at high-volume organizations (offering a donation incentive on survey completion) would produce direct causal evidence; another arm adding the certainty calibration question (Section 7.2) would calibrate hypothetical bias magnitude. We recommend adding the certainty question following the intent question, where possible.
• Referral weight validation. Adding the intent question to a referral-only survey can validate org-specific weights and improve reliable non-response estimation. Cross-tabulation results shared across the field also improve calibration for the ea_community weight (0.30), which currently rests on a single organization’s data, and other referral weights.
• Longitudinal panel for recurring donor CF. All current CF estimates are calculated from individual donors’ first donations and applied to their subsequent recurring donations. The interim decay framework we propose here has no empirical grounding. We recommend tagging the first-donation year in survey data to enable decay estimation once matched longitudinal data exists.

Acknowledgements

This report is a project of Rethink Priorities—a think-and-do tank dedicated to informing decisions made by high-impact organizations and funders across various cause areas. The author is Samara Mendez. Thank you to Melanie Basnak, David Moss, and William McAuliffe for their guidance, as well as Jamie Elsey for their helpful feedback. Thanks also to Urszula Zarosa and Elisa Autric for publishing the report online and assisting with dissemination.

We invite you to explore more RP research via our database and stay updated on new work by subscribing to our newsletter.

AI Transparency Statement

This report was written by Samara Mendez, with AI assistance (Claude Sonnet 5/Opus 5) at various stages of the coding, analysis, and writing process. To ensure data privacy and compliance, synthetic datasets were utilized during code development so that no personally identifiable information (PII) or raw donor records were shared with Claude. The report’s structure and logic were developed with minimal AI input; writing involved more AI assistance. All AI-assisted data analysis code and generated content were fully reviewed, edited, verified, and approved by humans.