The Instability of Philosophers' Judgments about Hypothetical Moral Scenarios

Non-philosophers' judgments about hypothetical philosophical scenarios are often labile and variable. They are subject, for example, to substantial order effects, and they vary by culture and gender. Question: Are professional philosophers' judgments about such hypothetical scenarios particularly better grounded? Jonathan Weinberg and Stephen Stich say no. Timothy Williamson says maybe yes. At stake, perhaps, is the reliability of "armchair philosophy" as a research method.

Fiery Cushman and I have started to examine the question experimentally. We begin by examining order effects on moral judgment: We presented hypothetical moral scenarios, in varying order, to three sets of respondents: professional philosophers, academic non-philosophers, and non-academics. (The professional philosophers had graduate degrees in philosophy and were mostly drawn from Leiter-ranked PhD-granting departments.) Non-philosophers, we suspected, would respond differently to the scenarios depending upon the order of presentation, a sign of instability and unreliability in judgment. Our question was: Would professional philosophers show smaller order effects and thus more stable judgments?

They did not. In fact, the overall trend across our data (marginally statistically significant) was for philosophers to show less stability their responses. This was true even for the subgroup of 91 respondents reporting a PhD in philosophy and a competence or specialization in ethics.

So, for example, the classic trolley problem comes in two versions. In the switch version, a bystander diverts a runaway trolley onto a sidetrack, killing one person on that sidetrack to save five people on the main track. In the push version, a bystander pushes a heavy person into the path of a runaway trolley, preventing the trolley from killing five people further down the track, but killing the heavy person. Now, whether they think such actions are good or bad, the majority of respondents rank the two actions the same when they are presented side-by-side. But order of presentation influences this result: Respondents are substantially more likely to rank the two actions the same when the push version is presented first than when the switch version is presented first. (Why? Intuitions about switch are unstable, and tend to descend to match push, if push is presented first. Intuitions about push are more stable, shifting around less, and thus often will not rise to match switch when switch is presented first.) Non-academics rated switch and push equivalently 53% of the time when switch was presented first and 68% of the time when push was presented first. For academic non-philosophers, the differential was 55% vs. 71%. For professional philosophers, it was 54% vs. 73%, including 50% vs. 75% for the ethics PhD subgroup -- effect sizes well within statistical chance of each other.

Moral luck scenarios (e.g., drunk driver hitting a tree vs. hitting a girl) and action-omission scenarios (e.g., snatching away a drowning person's life vest vs. failing to offer him a life vest) produced similar results. In each case, the order effects were about the same size for philosophers and non-philosophers. In our aggregate measure, philosophers (including ethics PhDs) trended toward showing slightly larger order effects overall than did the two comparison groups.

For further details, see our manuscript here, forthcoming in Mind & Language.