Survey on intermediate goals in AI governance

It seems that a key bottleneck for the field of longtermism-aligned AI governance is limited strategic clarity (see Muehlhauser, 2020, 2021). As one effort to increase strategic clarity, in October-November 2022, we sent a survey to 229 people we had reason to believe are knowledgeable about longtermist AI governance, receiving 107 responses. We asked about: 

• respondents’ “theory of victory” for AI risk (which we defined as the main, high-level “plan” they’d propose for how humanity could plausibly manage the development and deployment of transformative AI such that we get long-lasting good outcomes),
• how they’d feel about funding going to each of 53 potential “intermediate goals” for AI governance,^[1]
• what other intermediate goals they’d suggest,
• how high they believe the risk of existential catastrophe from AI is, and
• when they expect transformative AI (TAI) to be developed.

We hope the results will be useful to funders, policymakers, people at AI labs, researchers, field-builders, people orienting to longtermist AI governance, and perhaps other types of people. For example, the report could: 

• Broaden the range of options people can easily consider
• Help people assess how much and in what way to focus on each potential “theory of victory”, “intermediate goal”, etc.
• Target and improve further efforts to assess how much and in what way to focus on each potential theory of victory, intermediate goal, etc.

If you'd like to see a summary of the survey results, please request access to this folder. We expect to approve all access requests,^[2] and will expect readers to abide by the policy articulated in "About sharing information from this report" (for the reasons explained there).

Acknowledgments

This report is a project of Rethink Priorities–a think tank dedicated to informing decisions made by high-impact organizations and funders across various cause areas. The project was commissioned by Open Philanthropy. Full acknowledgements can be found in the linked "Introduction & summary" document. 

If you are interested in RP’s work, please visit our research database and subscribe to our newsletter. 

1. ^{^}

   Here’s the definition of “intermediate goal” that we stated in the survey itself: 

   By an intermediate goal, we mean any goal for reducing extreme AI risk that’s more specific and directly actionable than a high-level goal like ‘reduce existential AI accident risk’ but is less specific and directly actionable than a particular intervention. In another context (global health and development), examples of potential intermediate goals could include ‘develop better/cheaper malaria vaccines’ and ‘improve literacy rates in Sub-Saharan Africa’.

2. ^{^}

   If two days after you request access you still haven't received access, this is probably just due to a mistake or delay on our end, so please request access again.