I'm in favour of slowing the intelligence explosion (and in particular of "Pause at human-level".)

Shouldn't making that happen be a top priority, then? We as a civilization need to get a lot of groundwork in place to make a human-level pause happen: international agreement, defined red lines, enforcement mechanisms (e.g. GPU tracking), etc. Given how neglected these issues are and the catastrophic consequences of building ASI too quickly, it seems that laying the groundwork for a pause is a higher priority than, or at least on par with, the top causes on this post's "neglected cause areas" list.

I was reading the AI-enabled coups report and one of the mitigations is roughly "build the AI model such that it refuses to do coups". If you can pull that off then it solves the problem, but it means the model is incorrigible, and therefore you have to exactly specify the correct values up front because you're locked-in.

There might be some other report saying the way to avoid bad value lock-in is to make AI corrigible, and you can object to that by pointing out how it enables coups.

You can't look at the problems separately, you have to consider them at the same time and find a solution that works for every problem.

This is quite interesting; I hadn't thought of this. Do you think it should be approximated as "% chance that AI safety is actually bad" and "increase in AI risk per doubling of staff"?

I'm not sure. I would probably say that you shouldn't start a career in AI safety unless you can articulate a theory of why safety work has been harmful in the past, and how you're going to avoid more of the same. Building that theory is more important than adjusting the model inputs on a cost-effectiveness model.

This is a nice analysis and I wish more people would do things like this. A lot of the setup looks sensible to me. I thought it was clever how you used non-AI x-risk to determine the total number of people affected by AI x-risk, since this doesn't get you unwieldy Astronomical Waste-style numbers, but it also assigns positive value to future people. I'm not sure it's actually a good approach, but at least it's a clever idea.

It would be more useful to compare AI safety work vs. other longtermist interventions, since it's unlikely that donations to GiveWell would beat longtermist interventions from a longtermist POV. But I realize that would be a lot more work, and you've already put an admirable amount of work into this.

The biggest thing missing from the model is the possibility that safety research is net harmful. I believe much historical safety research ended up being harmful by making AI easier to commercialize and thus accelerating development (it would've been better for researchers to focus more on theoretical work that doesn't directly enable commercialization). I'm less sure about this but there may also be a replacement effect where empirical work on aligning current-gen models—which I don't think is very useful for aligning ASI—crowds out more important long-term work.

Alternatively, he believes that using extreme hyperbole is a good way to get attention.

I quickly read this post and Anil Seth's essay and I don't see the part where they argue for the thesis. I see various statements about how human brains work and about how computers work, but I don't see how they connect the dots to "...and therefore computers can't be conscious."

For example, the articles make the claim that brains make no clear separation between hardware and software. Okay, that seems to be true. But so what? Why should I believe that a lack of hardware/software distinction is a necessary property for consciousness to arise?

I feel like I'm missing a lot of what they're trying to say, but I also feel like that's the authors' fault, not mine, because the pieces (especially Seth's original essay) are structured in a way that makes it really hard for me to identify the central arguments.

I can't think of any reason off the top of my head why this would happen, except that you committed fraud.

When you have eliminated the impossible, whatever remains, however improbable, must be the truth.

I have eliminated the impossible by failing to think of any other hypotheses, therefore you must have committed fraud, and this spike is not real. I eagerly await either a failed replication causing you leave academia in disgrace, or your appointment to President of an elite university, followed one to two decades later by a resignation once someone finally gets around to doing a replication.

Cost-effectiveness is precisely the reason why I focus on AI safety. I can only speak for myself but I think the same is true for a lot of people. The thing that cuts against AI safety is more like "rigorously measurable cost-effectiveness", but that's not what I mean by "cost-effectiveness". You can't give a precise cost-effectiveness estimate for AI safety work, but it's pretty easy to show that it's orders of magnitude more cost-effective than GiveDirectly on any plausible set of assumptions.*

*unless it's net negative, which unfortunately much EA-adjacent AI safety work turned out to be...but at least we can say that it's orders of magnitude higher absolute impact

Jagged progress is conceivable, but it's virtually impossible that AI could replace all coders and accelerate math research but not replace other jobs, because coding and math research (specifically ML-type math) are exactly the skills needed to accelerate AI development. If AI can accelerate AI development, then the timeline to getting an AI that can replace humans on all tasks becomes much shorter.

I would've done something like that if I'd had any bread!