Formal Decision Theory Is an Optional Tool That Breaks When Values are Huge

Formal decision theory is a tool -- a tool that breaks, a tool we can do without, a tool we optionally deploy and can sometimes choose to violate without irrationality.  If it leads to paradox or bad results, we can say "so much the worse for formal decision theory", moving on without it, as of course humans have done for almost all of their history.

I am inspired to these thoughts after reading Nick Beckstead and Turuji Thomas's recent paper in Nous, "A Paradox for Tiny Probabilities and Enormous Values".

Beckstead and Thomas lay out the following scenario:

On your deathbed, God brings good news. Although, as you already knew, there's no afterlife in store, he'll give you a ticket that can be handed to the reaper, good for an additional year of happy life on Earth. As you celebrate, the devil appears and asks, ‘Won't you accept a small risk to get something vastly better? Trade that ticket for this one: it's good for 10 years of happy life, with probability 0.999.’ You accept, and the devil hands you a new ticket. But then the devil asks again, ‘Won't you accept a small risk to get something vastly better? Trade that ticket for this one: it is good for 100 years of happy life—10 times as long—with probability 0.999^2—just 0.1% lower.’ An hour later, you've made 10^50,000 trades. (The devil is a fast talker.) You find yourself with a ticket for 10^50,000  years of happy life that only works with probability .999^50,000, less than one chance in 10^21. Predictably, you die that very night. 

Here are the deals you could have had along the way:

[click image to enlarge and clarify]

On the one hand, each deal seems better than the one before. Accepting each deal immensely increases the payoff that's on the table (increasing the number of happy years by a factor of 10) while decreasing its probability by a mere 0.1%. It seems unreasonably timid to reject such a deal. On the other hand, it seems unreasonably reckless to take all of the deals—that would mean trading the certainty of a really valuable payoff for all but certainly no payoff at all. So even though it seems each deal is better than the one before, it does not seem that the last deal is better than the first.

Beckstead and Thomas aren't the first to notice that standard decision theory yields strange results when faced with tiny probabilities of huge benefits: See the literature on Pascal's Wager, Pascal's Mugging, and Nicolausian Discounting.

The basic problem is straightforward: Standard expected utility decision theory suggests that given a huge enough benefit, you should risk almost certainly destroying everything.  If the entire value of the observable universe is a googol (10^100) utils, then you should push a button that has a 99.999999999999999999999% chance of destroying everything as long as there is (or you believe that there is) a 0.00000000000000000000001% chance that it will generate more than 10^123 utils.

As Beckstead and Thomas make clear, you can either accept this counterintuitive conclusion (they call this recklessness) or reject standard decision theory.  However, the nonstandard theories that result are either timid (sometimes advising us to pass up an arbitrarily large potential gain to prevent a tiny increase in risk) or non-transitive (denying the principle that, if A is better than B and B is better than C, then A must be better than C).  Nicolausian Discounting, for example, which holds that below some threshold of improbability (e.g., 1/10^30), any gain no matter how large should be ignored, appears to be timid.  If a tiny decrease in probability would push some event below the Nicolausian threshold, then no potential gain could justify taking a risk or paying a cost for the sake of that event.

Beckstead and Thomas present the situation as a trilemma between recklessness, timidity, and non-transitivity.  But they neglect one horn.  It's actually a quadrilemma between recklessness, timidity, non-transitivity, and rejecting formal approaches to decision.

I recommend the last horn.  Formal decision theory is a limited tool, designed to help with a certain type of decision.  It is not, and should not be construed to be, a criterion of rationality.

Some considerations that support treating formal decision theory as a tool of limited applicability:

• If any one particular approach to formal decision theory were a criterion of rationality such that defying its verdicts were always irrational, then applying any other formal approach to decision theory (e.g., alternative approaches to risk) would be irrational.  But it's reasonable to be a pluralist about formal approaches to decision.
• Formal theories in other domains break outside of their domain of application.  For example, physicists still haven't reconciled quantum mechanics and general relativity.  These are terrific, well confirmed theories that seem perfectly general in their surface content, but it's reasonable not to apply both of them to all physical predictive or explanatory problems.
• Beckstead and Thomas nicely describe the problems with recklessness (aka "fanaticism") and timidity -- and denying transitivity also seems very troubling in a formal context.  Problems for each of those three horns of the quadrilemma is pressure toward the fourth horn.
• People have behaved rationally (and irrationally) for hundreds of thousands of years.  Formal decision theory can be seen as a model of rational choice.  Models are tools employed for a range of purposes; and like any model, it's reasonable to expect that formal decision theory would distort and simplify the target phenomenon.
• Enthusiasts of formal decision theory often already acknowledge that it can break down in cases of infinite expectation, such as the St. Petersburg Game -- a game in which a which a fair coin is flipped until it lands heads for the first time, paying 2^n, where n is the number of flips, yielding 2 if H, 4 if TH, 8 if TTH, 16 if TTTH, etc. (the units could be dollars or, maybe better, utils).  The expectation of this game is infinite, suggesting unintuitively that people should be willing to pay any cost to play it and also, unintuitively, that a variant that pays $1000 plus 2^n would be of equal value to the standard version that just pays 2^n.  Some enthusiasts of formal decision theory are already committed to the view that it isn't a universally applicable criterion of rationality.

In a 2017 paper and my 2024 book (only $16 hardback this month with Princeton's 50% discount!), I advocate a version of Nicolausian discounting. My idea there -- though I probably could have been clearer about this -- was (or should have been?) not to advocate a precise, formal threshold of low probability below which all values are treated as zero while otherwise continuing to apply formal decision theory as usual.  (I agree with Monton and Beckstead and Thomas that this can lead to highly unintuitive results.)  Instead, below some vague-boundaried level of improbability, decision theory breaks and we can rationally disregard its deliverances.

As suggested by my final bullet point above, infinite cases cause at least as much trouble.  As I've argued with Jacob Barandes (ch. 7 of Weirdness, also here), standard physical theory suggests that there are probably infinitely many good and bad consequences of almost every action you perform, and thus the infinite case is likely to be the actual case: If there's no temporal discounting, the expectation of every action is ∞ + -∞.  We can and should discount the extreme long-term future in our decision making much as we can and should discount extremely tiny probabilities.  Such applications take formal decision theoretical models beyond the bounds of their useful application.  In such cases, it's rational to ignore what the formal models tell us.

Ah, but then you want a precise description of the discounting regime, the thresholds, the boundaries of applicability of formal decision theory?  Nope!  That's part of what I'm saying you can't have.

Flipping Pascal's Wager on Its Head

In his famous Wager, Pascal contemplates whether one should choose to believe in God. (Maybe we can't directly choose to believe in God any more than we can simply choose to believe that the Sun is purple; but we can choose to expose ourselves to conditions, such as regular association with devoted theists, that are likely to eventually lead us to believe in God.) Although there's some debate about how exactly Pascal conceptualizes the decision, one interpretation is this:

Choose to believe: If God exists, infinite reward; if God does not exist status quo.

Choose to not to believe: If God exists, infinite punishment; if God does not exist status quo.

Suppose that your antecedent credence that God exists is some probability p strictly between 0 and 1. Employing standard decision theory, the expected payoff of believing is p * ∞ [the expected payoff if God does exist] + (1-p) * 0 [the expected payoff if God does not exist] = ∞. The payoff of not believing is p * -∞ + (1-p) * 0 = ∞. Since ∞ > -∞ (to put it mildly), belief is the rational choice.

Now maybe it's cheating to appeal to infinitude. Is Heaven literally infinitely good? (There might, for example, be diminishing returns on joyful experiences over time.) And maybe decision theory in general breaks down when infinitudes are involved (see my recent discussion here). But finite values also work. As long as the "status quo" value is the same in both conditions (or better in the belief condition than the non-belief condition), the calculus still yields a positive result for belief.

If not believing is better in the absence of God, it's a bit more complicated. (Non-belief might be better in the absence of God if believing truths is intrinsically better than believing untruths or if believing that God exists leads one to make sacrifices one wouldn't otherwise make.) But if Heaven would be as good as advertised, even a smidgen of a suspicion that God exists favors belief. For example, if life without belief is one unit better than life with belief, contingent on the non-existence of God, and if Heaven is a billion times better than that one-unit difference and Hell a billion times worse, then the expected payoff for believing in God is p * 1,000,000,000 + (1-p) * -1, and the expected payoff of not believing is p * -1,000,000,000 + (1-p) * 0. This makes belief preferable as long as you think the chance of God's existing is greater than about one in two billion.

So far, so Pascalian. But there's God and then there's the gods. It seems that a more reasonable approach to the wager would consider theistic possibilities other than Pascal's God. Maybe God is an adolescent gamer running Earth in a giant simulation. Maybe the universalists are correct and a benevolent God just lets everyone into Heaven. Or maybe a jealous sectarian God condemns everyone to Hell for failing to believe the one correct theological package (different from Pascal's).

If so, then the decision matrix looks something like this [click to enlarge and clarify]:

In other words, quite an un-Pascalian mess! If the positive and negative values are infinite, then we're stuck adding ∞ and -∞ in our outcomes, normally a mathematically undefined result. If the values are finite but large, then the outcome will depend on the particular probabilities and payoffs, which might be sensitive to hard-to-estimate facts about the total finite goodness of Heaven or badness of Hell. And of course even the decision matrix above is highly simplified compared to the range of diverse theistic possibilities.

But let me suggest one way of clarifying the decision. If God is not benevolent, all bets are off. Who knows what, if anything, an unbenevolent God might reward or punish? Little evidence on Earth points toward one vs another strategy for attaining a good afterlife under a hypothetical unbenevolent deity. I propose that we simplify by removing this possibility from our decision-theoretical calculus, instead considering the decision space on the assumption that if God exists God is benevolent. Doing that, we can get some decision-theoretic traction: a benevolent God, if he/she/it/they reward anything, should reward what's good.

This, then gives us mortals some (additional) reason to do whatever is good.

Here's something that's good: apportioning one's beliefs to the evidence. The world is better off, generally speaking, if people's credence that it will rain on Tuesday tends to match the extent of the evidence that it will rain on Tuesday. The world is better off, generally speaking, if people come to believe that cigarette smoking is bad for one's health once the evidence shows that, if people come to believe in anthropogenic climate change once the evidence shows that, if people decline to believe in alien abductions given that the evidence suggests against it, and so on. Apportioning our beliefs to the evidence is both a type of intellectual success that manifests the flourishing of our reasoning and a pragmatic path to the successful execution of our plans.

This is true for religious belief as well. Irrationally high credence in some locally popular version of God doesn't improve the world, but in fact has historically been a major source of conflict and suffering. Humanity would be better off without a tendency toward epistemically unjustified religious dogmatism. Nor should a benevolent God care much about being worshipped or believed in; that's mere vanity. A truly benevolent God, with our interests at heart, should care mainly that we do what is good -- and this, I suggest includes apportioning our religious beliefs to the evidence.

The evidence does not suggest that we should believe in the existence of God. (We could get into why, but that's a big topic! We can start by considering religious disagreement and the problem of evil.) If a benevolent God rewards or at least does not punish those who apportion their belief to tge evidence, a benevolent God should reward or at least not punish non-believers.

If God does not exist, we're better off apportioning our (non)belief to the (non)evidence. If a benevolent God exists, we're still better off not believing in the God. If God exists but is not benevolent, then decision-making policies break. Thus, we can flip Pascal's wager on its head: Unless we reject decision theory entirely as a means to evaluate the case, we're better off not believing than believing.