Friday, June 04, 2010

Ethicists' vs Non-Ethicists' Honesty in Questionnaire Responses

In 2009, Josh Rust and I ran a survey asking hundreds of ethicists, non-ethicist philosophers, and comparison professors, first, a variety of questions about their views on ethical matters (e.g., vegetarianism, voting, staying in touch with one's mother) and second, about their own personal behavior on the same matters. (No identifying information was associated with the responses, of course.) Some previous posts on the survey are here and here and here.

Now one of the cool things about this study is that in some cases we also have data on actual behavior -- thus enabling a three-way comparison of normative attitude, self-reported behavior, and actual behavior (as you'll see in the links above).

Thus, a measure of honesty falls out of the questionnaire: How well are the self-reports related to the actual behavior? Actually, we have two different types of measures of honesty: For most of the topics on which we didn't have direct behavioral data we asked two behavioral questions, one vague and easy to fudge, the other concrete and more difficult to fudge without explicit deceptive intent. So, for example, we asked at how many meals per week the respondent ate the meat of a mammal (the fudgy question) and also whether she ate the meat of a mammal at her last evening meal not including snacks (the concrete question). If, hypothetically, half the respondents who reported eating mammal meat at 3 or fewer meals per week also reported having eaten it at the last evening meal, we could infer that that group of respondents were fudging their answers.

We created a composite of six types of suspicious (or demonstrably false) responses and we compared the rates of suspicious responding between the groups. The six measures were:

* comparison of self-reported number of votes since the year 2000 with actual voting records;

* comparison of claims to have voted in the Nov. 2008 U.S. general Presidential election with actual voting records;

* comparison of claims of 100% or 95% responsiveness to undergraduate emails with responsiveness to emails that we had sent that were designed to look as though they were from undergraduates (see here, and yes we got IRB approval);

* for philosophers only, comparison of claims of membership in the American Philosophical Association with membership records (excluded from the analysis in this post, but discussed here);

* comparison of a general claim about how often the respondent talks with her mother (if living) with a specific claim about date of last conversation;

* comparison of a general claim about how often the respondent donates blood (if eligible) with a specific claim about date of last donation;

* comparison of a general claim about meals per week at which the respondent eats the meat of a mammal with a specific claim about the last evening meal.
We found that all three groups showed similar rates of suspicious responding: 50% of ethicists, 49% percent of non-ethicist philosophers, and 49% of comparison professors gave at least one suspicious response -- variation well within chance, of course, given the number of respondents. (Remember that on the second three measures a suspicious response is not necessarily a lie or even an unconscious self-serving distortion, but only an answer or pattern of answering that seems more likely to be false or distortive than another pattern would be, when aggregated across respondents.)

Thus, as in previous research we failed to find any evidence that ethicists behave any better than other socially comparable non-ethicists.

This is assuming, of course, that lying or giving distorted answers on surveys like ours is morally bad. Now, as it happens, we asked our respondents about that very issue, and 87% (89% of ethicists) said it was morally bad to answer dishonestly on surveys like ours. We also had a measure of how bad they thought it was -- a 9-point scale from "very morally good" through "morally neutral" to "very morally bad". As it turned out, there was no statistically significant relationship between normative attitude and rate of suspicious responding.

Near the end of the survey, we also explicitly asked respondents whether they had answered any of the questions dishonestly. Few said they did, and answers to this question appeared to be unrelated to rates of suspicious responding: Among respondents with no suspicious looking responses, 6 (2.2%) said they had answered dishonestly, compared to 7 (2.6%) of the respondents with at least one suspicious response.

Finally, we had asked the philosophy respondents whether they prefered a deontological, consequentialist, virtue ethical, or some other sort of normative ethical view. Deontologists are often portrayed as sticklers about lying -- Kant, the leading historical deontologist, was notoriously very strict on the point. However we detected no difference in patterns of suspicious responding according to normative ethical view. To the extent there was a trend, it was for the consequentialists to be least likely to have suspicious or false responses (47%, vs. 56% for deontologists and 58% for virtue ethicists; this analysis includes the APA question).

21 comments:

Autumnal Harvest said...

"This is assuming, of course, that lying or giving distorted answers on surveys like ours is morally bad. Now, as it happens, we asked our respondents about that very issue. . . Near the end of the survey, we also explicitly asked respondents whether they had answered any of the questions dishonestly."

Ah, but did you also ask them, after that, whether they had just dishonestly answered the question about answering questions dishonestly? And whether it would have been morally bad to do so for that specific question, as opposed to for surveys in general? ;)

Warren Davies said...

Very interesting. What do you think this means? Years of study of ethics does not necessarily make one more ethical? That's quite a surprising finding, you'd think that even by priming alone the ethicists would behave more ethically.

Eric Schwitzgebel said...

Autumnal: Yes, it's all pretty loopy and meta-! So who knows what exactly to make of that question. I thought it would be interesting to ask, though.

Eric Schwitzgebel said...

Warren: yes, that's what our various results so far suggest. I'm glad to hear you find it surprising. I do too, in a way. Some people seem to think it obvious that ethicists would not behave better. I find that view a bit too confidently cynical.

Warren Davies said...

I do find it surprising. Why do people think it is obvious that ethicists wouldn't behave better? Is it something to do with the difference between learning something 'in theory', so to speak, versus in practice?

Do you know anything about who actually does behave more ethically? A few studies on altruism and prosocial behaviour spring to mind (e.g., CCTV encourages prosocial behaviour if I remember rightly). But that's not quite the same as integrating a set of ethics and adhering to them. The complete opposite, in a way.

Protagoras said...

It seems to be a common view in the profession (at least among non-deontologists) that deontologists are a bit dodgy, while consequentialists are mostly not. I assume that the consequentialist's 47% suspicious response rate wasn't low enough to be significant, but I do think comparing the different kinds of ethicists is very worthwhile (and as a consequentialist, I'm happy to see my suspicions get a tiny bit of support).

Virtue ethicists don't have the dodgy rep that deontologists do, so I'm surprised that they turned out even worse, but again presumably the effect is too small to be significant. More data is clearly needed.

Eric Schwitzgebel said...

Thanks for the comments, folks!

Warren: My guess is that it's not very well worked out -- some combination of personal experience, hindsight bias, and cynicism (cf cynicism about priests and moralizing politicians), then when I ask people for theoretical grounding, they reach around for any plausible story.

Eric Schwitzgebel said...

Aaron, your view appears to be a common one. (Josh and I have a footnote about it in our "Peer Opinion" paper.) Next week, after I have done a few more analyses, I hope to have a post on it.

Eric Schwitzgebel said...

Warren, ps, on your other point: high ses people commit fewer crimes, it's said. There is also a methodologically dubious literature that might or might not be interpreted as suggestion a negative relationship between religiosity and crime or "deviance".

Unknown said...

The more you do these surveys, the more I wonder if just doing them might itself lead to distortions-- just last week, I received an email from a student I didn't recognize by name (or otherwise) demanding to know why Ayn Rand isn't taught in philosophy classes. I politely answered the thing, suggesting he take some intro classes with people of whose intros I think particularly well. Then I hit send.
Then I paused for a moment, and found myself thinking-- I wonder if this Eric S or one of those x-phi guys testing to see how ethicists respond to Randians...
with smiles,

J.Vlasits said...

Maybe your data suggests something that was first pointed out (as far as I know) by Aristotle in the Nichomachean Ethics. He says in book I that the subjects of ethics and politics should not be taught to the young (or in fact, to ones with a "youthful disposition") because it concerns action and not knowledge. And the young, like the incontinent are too much guided by their desires. Ethical study, he says, can only benefit those who themselves are already generally good.

I wonder if your results suggest something like incontinence among moral philosophers which may not necessarilly be evidence against the value of moral theorizing but rather be evidence that many philosophers are simply not the kinds of people who can benefit from moral theorizing. Neither interpretation seems particularly optimistic!

Eric Schwitzgebel said...

Yes, I do wonder about that, Kate. I've had a few people email me asking about odd emails they have received. I hope and expect that we won't be doing the email test again!

Eric Schwitzgebel said...

J.: Yes, Aristotle is interesting on moral development. Since he wants the philosophical study of ethics to help us become good, it would be frustrating indeed if philosophers themselves were just the sort of people who couldn't benefit from it!

Autumnal Harvest said...

Warren, I'll defend the view that the results are "obvious," or at least "not surprising," despite the fact that it's somewhat bad manners to characterize someone's research findings as "obvious."

It seems to me that for most ethical situations that people face in their everyday lives, it's not difficult to know what the ethical thing to do is, and it's not important, in practice, to have a solid theoretical foundation explaining why it's the ethical thing to do - it's just that they don't want to do the ethical thing, because the ethical course involves unpleasant costs. For example, most professors not trained in ethics know perfectly well that they should answer students' e-mails, and while an ethics professor may (or may not) be better able explain why that's the case, that explanation is largely irrelevant to the practical conflict, which is (1) I know should answer students' e-mails but (2) I have other things I want to spend my time on. How does studying ethics make you more likely to resolve the conflict between (1) and (2) in favor of (1)?

Eric Schwitzgebel said...

That's nicely put, Autumnal. A couple possibilities: People drawn to teach and research ethics might be the sort of people who care more than average about ethics, and if so it's not unreasonable to think they might resolve the conflict in the moral direction more often than average. Or moral considerations might be more salient to them, might come explicitly to mind more often, and thereby have more motivational power. I'm not saying either of these is true -- just that they're not obviously false.

Protagoras said...

That's a good point. It might be better to look at where ethicists stand on controversial issues (though again comparing what they say to what they do in their lives, and not just looking at what they say). Those are the sorts of things where one might expect ethical expertise to help. Of course, there's the difficulty of determining what to count as right on controversial ethical questions, but nobody said social science research was easy. Or at least nobody should have said that.

Eric Schwitzgebel said...

Agreed on all points, Aaron. One controversial issue where I do have data is vegetarianism. Ethicists are vastly more likely than professors in departments other than philosophy to say eating the meat of mammals is morally wrong, but their differences in behavior are subtle if any. (You can find my old post on this by searching within the blog.)

Autumnal Harvest said...

Good points, Eric. I guess I wouldn't say that the results are "obvious," but rather that I'm not surprised by them. "Not surprised" doesn't mean that they're not interesting, though.

The results on meat-eating are particularly interesting! Perhaps it's a matter of expectations, but I actually think they makes ethicists look good - or, at least, better than I would have expected. It looks like ethics professors are about twice as likely to totally abstain from eating meat as non-philosophy professors. Did any of the professors who reported 0 mammal-meals per week also report eating mammals at their last dinner? If not, I think it would be reasonable to take the total abstainers at their word (since totally abstention is also difficult to fudge), and let the ethicists claim a victory here. Trying to quit a bad/immoral habit by just toning it down is pretty difficult; I think cold turkey quitting is a good dividing standard here.

Eric Schwitzgebel said...

Autumnal: It's odd that the meals/week seem to come apart from the last evening meal results. The last evening meal results have the disadvantage of being relatively noisy and low-power but the meals/week results have the disadvantage of inviting fudging, so it's not straightforward which to take more seriously.

I would agree that it's a reasonable guess, a priori, that 0 meals/weak would be relatively immune to fudging: Either one is a strict vegetarian or not. Of the 112 respondents who claimed 0 meals/week, two said they had eaten the meat of a mammal at their last evening meal. It is possible they just hit the wrong button. However, if we assume that they are accurately reporting, and if we assume that about half of mammal-meat meals are evening (vs. lunch, breakfast, and snacks combined), and finally if we assume that the two 0/yes responders are infrequent mammal meat eaters rather than just bald liars on the meals/week question who eat meat many nights, then we can estimate that about 28 (25%) of the respondents who said they ate 0 meals/week actually at one meal -- which suggests a non-trivial rate of fudging. Also, it might underestimate the rate of fudging, since some poeple might resist answering that they had eaten it at the last evening meal after having reported zero/week to the previous question. If there is any question where respondents will be motivated to flat dishonesty, it might be that one.

Given all this, I interpret the data as suggesting that ethicists show only a weak tendency, if any, to eat less mammal meat than do the other groups, despite the large difference in normative view.

Autumnal Harvest said...

On the one hand, I want to agree with your analysis, because it agrees with my preconception that ethicists will behave no better than anyone else. On the other hand, I'm not sure 2 out of 112 mammal-meat meal responses is enough give up my a priori belief that the 0 meals/week responses are largely free of fudging. It seems to me that surveys with enough data ar generally going to have logical inconsistencies on the scale of 5% or more if you dig into them enough. For example, the most recent Pew survey on religion I saw had something like 6% of atheists being certain that there was a personal god, and 10% believing in an afterlife. I think there's always enough randomness, due to people not understanding the questions, clicking buttons at random, and who knows what else, that I don't feel comfortable extrapolating anything from a 2% discrepancy.

I guess I can just conclude with the old standby: "More data is needed." There! No one can disagree with that. :)

Eric Schwitzgebel said...

I'm inclined to agree, Autumnal, that 2 is margin of error from 0. But it's also not confirmatory of a lack of fudging for the reasons I just mentioned. So the degree of fudging will have to be decided on other grounds.