Scott Alexander argued that we should rule some things (e.g. thinkers) in, not out. Ord et al (2010) explain how, when a model or argument seeks to establish a tiny probability as the best estimate of some high-stakes event (e.g. human extinction), the estimated probability may be “dwarfed by the chance that the argument itself is flawed.” Putting these together yields an important methodological heuristic for thinking about longtermism: given our immense uncertainty about the future, and the wide range of possible models and scenarios over which we should remain reasonably uncertain, we should rule high stakes in, not out.
Even just a 1% chance of extremely high stakes is sufficient to establish high stakes in expectation. So we should not feel assured of low stakes even if a highly credible model—warranting 99% credence—entails low stakes. It hardly matters at all how many credible models entail low stakes. What matters is whether any credible model entails extremely high stakes. If one does—while warranting just 1% credence—then we have established high stakes in expectation, no matter what the remaining 99% of credibility-weighted models imply (unless one inverts the high stakes in a way that cancels out the other high-stakes possibility).
In what follows, I’ll step through some highlights from the Essays on Longtermism collection that hit differently with this methodological dictum in mind.
Ruling-in Longtermist Causes
‘Minimal and Expansive Longtermism’, by Greaves & Tarsney, contrasts a “minimal” view—on which promising longtermist projects are basically exhausted by a handful of x-risk reduction priorities—from a more “expansive” view on which “There is a very wide variety of ways to greatly improve the expected value of the far future.” (And this is apt to remain true for the foreseeable future, almost no matter how many resources are dedicated to longtermism.)
They write:
The problem is that, in general, the project of trying to influence the course of the far future has a strong air of intractability. The further into the future we look, the harder it becomes to predict either what the world will look like in the absence of any intervention on our part, or the effects of any particular present action. Risks of human extinction and other ‘existential catastrophes’ create an exception to these worries about intractability, since each such risk comes with a strong and clear ‘lock-in’ mechanism. But most other ways in which we might hope to improve the far future of humanity can be motivated only via significantly more speculative reasoning concerning very long-term causal chains.
This all seems true, but may not matter so much if the “ruling in” heuristic is apt. For example, the authors later report being “unconvinced” by Tyler Cowen’s arguments for compounding economic growth as a longtermist cause. Granted, it seems perfectly reasonable to doubt that economic growth could both (i) continue so long, and (ii) continue to have significant marginal effects on well-being all the while. But that isn’t determinative; what instead matters is whether we should grant some credence to Cowen’s model of the world. (And I suspect we should!)
Later in the paper, the authors write:
Our impression is that some longtermists believe that the marginal value of the best longtermist interventions exceeds the neartermist benchmark by 10:1 or less, while others believe that this ratio is 10,000:1 or more. And there is more than enough reasonable basis for this wide range of views. In particular, estimates of the expected future population that would be lost to existential catastrophe span tens of orders of magnitude.
I wonder how much of this disagreement is due to people offering different best-guess models rather than “all-models-considered” expected values, where low-end estimates require giving near-zero credence to models implying vastly higher stakes.
Overall, my sense is that the case for “expansive longtermism” may be greatly aided by the “rule high stakes in, not out” principle, together with the possibility of credible-seeming (albeit highly speculative) arguments for a wide range of potential longtermist interventions other than traditional x-risk reduction.[1]
Changing Population Levels
I especially liked the paper, ‘Is Extinction Risk Mitigation Uniquely Cost-Effective? Not in Standard Population Models’ by Gustav Alexandrie & Maya Eden. They note that standard models “typically imply that any shocks that proportionally decrease all factors of production have proportional, permanent effects on long-run population levels.” This means that disasters (like asteroids or wars) that destroy capital in addition to causing deaths (short of extinction) may be vastly more harmful than a comparably deadly pandemic that leaves capital intact (leading to permanently greater population recovery). Conversely, “the combined intervention of saving a life and increasing the capital stock to offset the decline in the capital-labor ratio” may, if standard models are correct, result in “a permanent, proportional increase in the population.” Given various “extremely simplifying assumptions”, this population-boosting intervention could even turn out to be more cost-effective than standard x-risk reduction interventions.[2]
The authors responsibly caution against taking their calculations at face value, given all the questionable assumptions involved. Their aim is rather “to illustrate, using an empirically grounded example, that there may indeed be interventions other than extinction risk mitigation that are cost-effective in virtue of having a long-run effect on the size of the global population.” Given my rule high stakes in principle, we should take this possibility seriously so long as we can’t rule out their assumptions (even if we consider the assumptions in question very likely mistaken).
Cluelessness and the Optimism-Pessimism Dilemma
In ‘Longtermist Myopia’, Askell & Neth argue that:
even if you accept that the future matters just as much as the present from a moral point of view, there are important reasons to focus on the near-term consequences of our actions for the purpose of decision-making. These reasons include… epistemic diffusion of our action’s predictable consequences, and both optimism and pessimism about existential risk…
“Epistemic diffusion” is the claim that effects get progressively harder to predict with temporal distance, until after a certain point we’re completely clueless about the far future. The authors mention how “chaotic systems” magnify “small initial uncertainties” until the far future is essentially unpredictable no matter what present actions we perform.
But note that this is just one model of our epistemic situation. Other models might imply that we are merely uncertain, not completely clueless. We might have more reason to expect present population loss to result in a smaller rather than a larger long-term population, for example. We might expect that present economic growth is more likely to increase than to decrease long-term economic growth. We might expect that robust education and good-government norms are more likely to be good than bad for the future of humanity, while a global nuclear war would be the opposite. And so on. One can certainly imagine scenarios in which any one of these expectations turns out to be mistaken. But I take it that they would be surprising results.
Moreover, I take it that there is very little credibility to the opposite view, that we should regard the inverse of the above claims as disproportionately likely by default. So if you give some (higher-order) credence to views or models implying cluelessness, and some to views on which we can often reasonably expect commonsensically good things to be long-term good, then it seems the positive expectations could trivially win out. The possibility of cluelessness seemingly “drops out” as deliberatively inert, compared to any more epistemically committing views that we also take seriously. (Compare similar arguments for the deliberative irrelevance of moral nihilism.)
So again, rather than allowing the possibility of cluelessness to rule out high (expectational) stakes, it seems we should take the opposing possibility—of being “clued in” to high long-term stakes—to rule in the high stakes of reasonable longtermist efforts.
Next, the authors discuss the “optimism-pessimism dilemma” (inspired by Thorstad 2022). On “optimistic” models, we’ll survive by default so x-risk reduction is a waste of time and resources. On “pessimistic” models, we can’t permanently reduce the risks enough for the efforts to be worth much: avoiding one extinction event just saves us a few years until the next one strikes—far from the vast glorious future that longtermists imagine. It’s only in the sweet spot that x-risk reduction efforts deliver on their potential for “astronomical value”.
You can probably anticipate my response by now: we only need one credible model entailing extremely high stakes in order to establish high stakes in expectation. And “credible” here does not even require high credence.
Compare Thorstad’s classic argument:
I show that pessimism is unlikely to ground the overwhelming importance of existential risk mitigation unless it is coupled with an empirical hypothesis: the time of perils hypothesis. However, I argue, the time of perils hypothesis is probably false. I conclude that existential risk pessimism may tell against the overwhelming importance of existential risk mitigation.
His paper’s conclusion more modestly calls the arguments for the time of perils hypothesis “inconclusive”. But either way, the time of perils hypothesis can (and should) rationally shape our expected value judgments without needing to be conclusively established or even probable. Warranting some non-negligible credence would suffice. Because, again, even just a 1% chance of extremely high stakes establishes high stakes in expectation.
The way expected value works, high stakes can get ruled in by a single credible scenario (absent a counterbalancing case with values inverted), and only ruled out by establishing that there is no way to rule them in. To seek to establish low expected value via appeal to a single “best guess” model or scenario is tempting but ultimately misguided. To rule out high stakes, you need to establish that the most longtermist-friendly scenario or model is not just unlikely, but vanishingly so.[3]
- ^
In particular, I take it that there are many candidate interventions whose potential for vast positive value is not counterbalanced by a comparable risk of vast negative value. We can thus attribute vastly positive expected value—while we face immense uncertainty, we are not completely clueless about what sorts of things are apt to be good for the longterm future.
- ^
This assumes a simple total axiology for valuing lives. Some may wish to give extra moral weight to avoiding outright unacceptable outcomes (like early extinction) compared to “luxury benefits” like increasing population further above a generally acceptable-seeming level.
- ^
To be clear, people are perfectly entitled to assign vanishingly low credence to claims when they sincerely believe it warranted—I certainly don’t mean to rule this out as a matter of form. (It’s how I’m inclined to dismiss Pascalian muggers, after all.) I take the sorts of scenarios discussed above to be, despite their very uncertain and speculative nature, vastly more substantively credible than the claims of Pascalian muggers. But there’s certainly plenty of room for further substantive dispute about what credences are really warranted in all these cases.
I don't think this works, at least at the level of our empirical credences, for reasons I argue here. (I think the crux here is the "insensitivity to mild sweetening" of imprecise expectations / incomplete preferences; more on that here.)
I do think what you say might work as a response to the precise Bayesian epistemic diffusion model, to be clear, but that's not the strongest case for cluelessness.
This seems true and useful to me, I'm surprised at the low agreement and karma scores!
I discuss another example here, where (using your framing) we cannot rule out that we are in the hinge of history, and since the stakes would then be so high, we ought to act significantly on that basis.
Interested if you agree with this example.
Thanks, yeah, I like your point there that "false negatives are costlier than false positives in this case", and so even <50% credence can warrant significant action. (I wouldn't literally say we should "act as if 3H is true" in all respects—as per Nuno's comment, uncertainty may justify some compounding "patient philanthropy", which could have high stakes if the hinge comes later. But that's a minor quibble: I take myself to be broadly in agreement with your larger gist.)
Edit: I no longer endorse this any more. The important point I was missing was that Person A's probability of extinction per century only needs to decay as 1/N in order for the value of the future to remain enormous, and a 1/N decay is not implausibly overconfident.
You are saying that we do not need to assign high probability to the "time of perils" hypothesis in order to get high stakes. We only need to assign it non-vanishing probability. And assigning it vanishing probability would appear to be implausibly overconfident.
But I'm not sure this works, because I think it is impossible to avoid assigning vanishingly small probability to some outcome. If you just frame the question differently, the position that appears to be the overconfident one can be reversed.
Suppose you ask two people what credence they each have in the "time of perils" hypothesis. Person A replies with 10%, and Person B replies with 10^(-20). Person B sounds wildly overconfident.
But now ask each of them what the probability is that humanity (or humanity's descendants/creations) will go extinct in the 50th century, conditional on surviving until that point. Person B may respond in many different ways. Maybe they say 1/1000. But Person A is now committed to giving a vanishingly small answer to this question, in order to be consistent with their 10% credence in "time of perils". Now it is Person A who sounds overconfident!
Person A is committed to this, because Person A places a non-vanishing probability on the future being very large. But the probability of making it to the far future is just the product of the probabilities of making it through each century along the way (conditional on surviving the centuries prior). For there to be a non-vanishing probability of a large future, most of these probabilities must be extremely close to 1. Does that not also seem overconfident?
I don't think this example tells us which of Person A or Person B are doing the right thing, but I think it shows that we can't decide between them with an argument of the form: "this person's view is implausible because they are assigning vanishingly small probability to something that seems, on the face of it, credible".
I'm not seeing the barrier to Person A's thinking there's a 1/1000 chance, conditional on reaching the 50th century, of going extinct in that century. We could easily expect to survive 50 centuries at that rate, and then have the risk consistently decay (halving each century, or something like that) beyond that point, right?
If you instead mean to invoke, say, the 50 millionth century, then I'd think it's crazy on its face to suddenly expect a 1/1000 chance of extinction after surviving so long. That would no longer "seem, on the face of it, credible".
Am I missing something?
I was assuming in my example that the "Time of perils" that Person A believes we might be living through is supposed to be over by the 50th century, so that the 50th century is already in the period where extinction risk is supposed to have become very low.
But suppose Person A adopts your alternative probabilities instead. Person A now believes in a 1/1000 chance of going extinct in the 50th century, conditional on reaching it, and then the probability halves in each century after that.
But if that's what they believe, you can now just run my argument on the 100th century instead. Person A now proposes a probability of ~10^(-18) of going extinct in the 100th century (conditional on reaching it) which seems implausibly overconfident to me on the face of it!
I agree with you, that if we were considering the 50 millionth century, then a probability of 1/1000 would be far too high. I agree that it would be crazy to stipulate a probability for the Nth century that is much higher than 1/N, because surviving N centuries is evidence that typical extinction risk per century is lower than this (except maybe if we were considering centuries close to the time the sun is expected to die..?)
But my point is that in order to get a truly big future, with the kind of stakes that dominate our expected value calculations, then we need the probability of extinction to decay much faster than 1/N. We need the "Time of Perils" hypothesis. It needs to decay exponentially* (something like the halving that you've suggested). And before too long that exponential decay is going to lead to implausibly low probabilities of extinction.
*Edit: Actually not too confident on this claim now I think it through some more. Perhaps you can still get a very large future with sub-exponential decay. Maybe this is another way out for Person A in fact!
Having thought this through some more, I've realised I'm wrong, sorry!
Person A shouldn't say that the probability of extinction halves each century, but they can say that it will decay as 1/N, and that will still lead to an enormous future without them ever seeming implausibly overconfident.
A 1/N decay in extinction risk per century (conditional on making it that far) implies a O(1/N) chance of surviving >= N centuries, which implies a O(1/N^2) chance of going extinct in the Nth century (unconditional). Assuming that the value of the future with extinction in the Nth century is at least proportional to N (a modest assumption) then the value of the future is the sum of terms that decay no faster than 1/N, so this sum diverges, and we get a future with infinite expected value.
I think your original argument is right.
I still have separate reservations about allowing small chances of high stakes to infect our decision making like this, but I completely retract my original comment!
Thanks for looking into it more!