I recently wrote 80k’s new problem profile on extreme power concentration (with a lot of help from others - see the acknowledgements at the bottom).

It’s meant to be a systematic introduction to the risk of AI-enabled power concentration, where AI enables a small group of humans to amass huge amounts of unchecked power over everyone else. It’s primarily aimed at people who are new to the topic, but I think it’s also one of the only write-ups there is on this overall risk,^[1]so might be interesting to others, too.

Briefly, the piece argues that:

• Automation could concentrate the power to get stuff done, by reducing the value of human labour, empowering small groups with big AI workforces, and potentially giving one AI developer a huge capabilities advantage (if there’s an intelligence explosion).
• This could lead to unprecedented concentration of political power via some combination of:
  • Humans deliberately seizing power for themselves (as with AI-enabled coups)
  • Some people becoming obscenely wealthy, such that government incentives are distorted in their favour or they simply outgrow the rest of the world
  • The erosion of people’s ability to understand what’s going on and coordinate in their own interests (either through deliberate interference by powerful actors, or more emergent dynamics)
• AI-enabled power concentration could cause enormous and lasting harm, by disempowering most people politically, and enabling large-scale abuses of power.
• There are ways to reduce the risk, but very few are working on them.

That’s my best shot at summarising the risk of extreme power concentration at the moment. I’ve tried to be balanced and not too opinionated, but I expect many people will have disagreements with the way I’ve done it. Partly this is because people haven’t been thinking seriously about extreme power concentration for very long, and there isn’t yet a consensus way of thinking about it. To give a flavour of some of the different views on power concentration:

• Some people like to think of human power concentration as a distinct risk from AI takeover. Others don’t think that distinction is particularly meaningful,^[2]and think of both human and AI takeover as forms of power concentration.
• Some people are mostly worried about scenarios where one or a small number of humans end up in power; others are also worried about hundreds or thousands of people having unchecked power.
• Some people are mostly worried about power-seeking humans deliberately seizing power, others are more worried about economic forces and incentives empowering the few, even if they’re not deliberately aiming for power.
  • Among those who are mostly worried about deliberate power-seeking, some people are mostly worried about a lab CEO taking over the world, some people are mostly worried about a head of state doing so.

So you shouldn’t read the problem profile as an authoritative, consensus view on power concentration - it’s more a waymarker, my best attempt to give an interim overview of a risk which I hope we will develop a much clearer understanding of, hopefully soon.

Some salient things about extreme power concentration that I wish we understood better:

• The relative importance of a) powergrabs, b) gradual disempowerment dynamics, c) intelligence curse dynamics, when it comes to combatting the risk of extreme power concentration.^[3]
  • People are often working on interventions that only help with one of those threat models. Maybe that’s fine, because they happen to all be similarly important. But maybe one is much more important than the others, or there are cross-cutting interventions that help with all of them and are more robust to uncertainty (I think transparency into who is using which AI capabilities with how much compute is a good candidate here). I’d like there to be more analysis of the relative importance of these different threat models, how they interact, and the best ways to intervene on them.
• What’s going on with the epistemics part
  • One of the things we included in the piece was 'epistemic interference', where the ability of most people to understand what’s happening and coordinate in their own interests gets eroded. I think this might be a super important dynamic, and might have an early-ish point of no return: if we lose the ability to sense-make, we’re probably going to lose all of the other games, too.
  • But I found surprisingly little analysis of how epistemic interference could happen concretely, how big a deal it is, or how we could stop it. I hope I just failed to find all the great existing work on this; I think it’s more likely that there just isn’t much on this topic so far, and think that more work here should be a high priority.
• What’s bad about extreme power concentration (though this one feels less pressing to me personally)
  • Though there are also other arguments, what I’m personally most compelled by at the moment is some combination of common sense (‘sounds bad!’) and aesthetics (‘I just don’t like the feel of a universe with one single dominant actor’). But that is very woolly.
  • There are also galaxy-brained arguments that power concentration is fine/good (because it’s the only way to stop AI takeover, or because any dictator will do moral reflection and end up pursuing the good regardless).
  • And galaxy-brained arguments that it’s actually bad after all (because if you only have a single dictator, they might just be very idiosyncratic or fail to do moral reflection and most of the value of the future will be lost).
    • If this argument is right, then powergrabs look like the most important route to power concentration (as it’s difficult to get to one person in charge without them). Though even then, it’s not clear whether the best place to intervene to prevent powergrabs is late stage, when it looks more like a powergrab, or early stage, when it looks more like the gradual erosion of checks and balances.

(For more musings on power concentration, you can listen to this podcast, where Nora Ammann and myself discuss our different takes on the topic.)

If you have thoughts on any of those things, please comment with them! And if you want to contribute to this area, consider:

• Reading the problem profile, or sharing it with people you think would benefit from reading it
• Expressing your interest in reducing extreme power concentration^[4]

Thanks to Nora Ammann, Adam Bales, Owen Cotton-Barratt, Tom Davidson, David Duvenaud, Holden Karnofsky, Arden Koehler, Daniel Kokotajlo, and Liam Patell for a mixture of comments, discussion, disagreement, and moral support.

1. I think AI-enabled coups, gradual disempowerment and the intelligence curse are the best pieces of work on power concentration so far, but they are all analysing a subset of the scenario space. I’m sure my problem profile is, too - but it is at least trying to cover all of the ground in those papers, though at a very high level. ↩︎

2. A few different complaints about the distinction that I’ve heard:

   • Most takeover scenarios will involve both human and AI power-seeking, so it will look blurry at the time
   • Even if it eventually becomes clear that either a human or an AI is in charge, it’s not particularly important - either way we’ve lost most of the value of the future ↩︎

3. (This is just an opportunistic breakdown based on the papers I like. I’d be surprised if it’s actually the best way to carve up the space, so probably there’s a better version of this question.) ↩︎

This is a form run by Forethought, but we’re in touch with other researchers in the power concentration space and intend to forward people on where relevant. We’re not promising to get back to everyone, but in some cases we might be able to help with funding, mentorship or other kinds of support. ↩︎

Show all footnotes