Wednesday, January 07, 2015

Psychology Research in the Age of Social Media

Popular summaries of fun psychology articles regularly float to the top of my Facebook feed. I was particularly struck by this fact on Monday when these two links popped up:

The links were striking to me because both of the articles reported studies that I regarded as methodologically rather weak, though interesting and fun. It occurred to me to wonder whether social media might be seriously aggravating the perennial plague of sexy but dubious psychological research.

Below, I will present some dubious data on this very issue!

But first, why think these two studies are methodologically dubious? [Skip the next two paragraphs if you like.]

The first article, on whether the rich are jerks, is based largely on a study I critiqued here. Among that study's methodological oddities: The researchers set up a jar of candy in their lab, ostensibly for children in another laboratory, and then measured whether wealthy participants took more candy from the jar than less-wealthy participants. Cute! Clever! But here's the weird thing. Despite the fact that the jar was in their own lab, they measured candy-stealing by asking participants whether they had taken candy rather than by directly observing how much candy was taken. What could possibly justify this methodological decision, which puts the researchers at a needless remove from the behavior being measured and conflates honesty with theft? The linked news article also mentions a study suggesting that expensive cars are more likely to be double-parked. Therefore, see, the rich really are jerks! Of course, another possibility is that the wealthy are just more willing to risk the cost of a parking ticket.

The second article highlights a study that examines a few recently-popular measures of "individualistic" vs. "collectivistic" thinking, such as the "triad" task (e.g., whether the participant pairs trains with buses [because of their category membership, supposedly individualistic] or with tracks [because of their functional relation, supposedly collectivistic] when given the three and asked to group two together). According to the study, the northern Chinese, from wheat-farming regions, are more likely to score as individualistic than are the southern Chinese, from rice-farming regions. A clever theory is advanced: wheat farming is individualistic, rice farming communal! (I admit, this is a cool theory.) How do we know that difference is the source of the different performance on the cognitive tasks? Well, two alternative hypotheses are tested and found to be less predictive of "individualistic" performance: pathogen prevalence and regional GDP per capita. Now, the wheat vs. rice difference is almost a perfect north-south split. Other things also differ between northern and southern China -- other aspects of cultural history, even the spoken language. So although the data fit nicely with the wheat-rice theory, many other possible explanations of the data remain unexplored. A natural starting place might be to look at rice vs. wheat regions in other countries to see if they show the same pattern. At best, the conclusion is premature.

I see the appeal of this type of work: It's fun to think that the rich are jerks, or that there are major social and cognitive differences between people based on the agricultural methods of their ancestors. Maybe, even, the theories are true. But it's a problem if the process by which these kinds of studies trickle into social media has much more do to with how fun the results are than with the quality of the work. I suspect the problem is especially serious if academic researchers who are not specialists in the area take the reports at face value, and if these reports then become a major part of their background sense of what psychological research has recently revealed.

Hypothetically, suppose a researcher measured whether poor people are jerks by judging whether people in more or less expensive clothing were more or less likely to walk into a fast-food restaurant with a used cup and steal soda. This would not survive peer review, and if it did get published, objections would be swift and angry. It wouldn't propagate through Facebook, except perhaps as the butt of critical comments. It's methodologically similar, but the social filters would be against it. I conjecture that we should expect to find studies arguing that the rich are morally worse, or finding no difference between rich and poor, but not studies arguing that the poor are morally worse (though they might be found to have more criminal convictions or other "bad outcomes"). (For evidence of such a filtering effect on studies of the relationship between religion and morality, see here.)

Now I suspect that in the bad old days before Facebook and Twitter, popular media reports about psychology had less influence on philosophers' and psychologists' thinking about areas outside their speciality than they do now. I don't know how to prove this, but I thought it would be interesting to look at the usage statistics on the 25 most-downloaded Psychological Science articles in December 2014 (excluding seven brief articles without links to their summary abstracts).

The article with the most views of its abstracted summary was The Pen Is Mightier Than the Keyboard: Advantages of Longhand over Laptop Note Taking. Fun! Useful! The article had 22,389 abstract views in December, 2014. It also had 1,320 full text or PDF downloads. Thus, it looks like at most 6% of the abstract viewers bothered to glance at the methodology. (I say "at most" because some viewers might go straight through to the full text without viewing the abstract separately. See below for evidence that this happens with other articles.) Thirty-seven news outlets picked up the article, according to Psychological Science, tied for highest with Sleep Deprivation and False Memories, which had 4,786 abstract views and 645 full text or PDF downloads (at most 13% clicking through).

Contrast these articles with the "boring" articles (not boring to specialists!). The single most downloaded article was The New Statistics: Why and How: 1,870 abstract views, 4,717 full-text and PDF views -- more than twice as many full views as abstract views. Psychological Science reports no media outlets picking this one up. I guess people interested in statistical methods want to see the details of the articles about statistical methods. One other article had more full views than abstract views: the ponderously titled Retraining Automatic Action Tendencies Changes Alcoholic Patients’ Approach Bias for Alcohol and Improves Treatment Outcome: 164 abstract views and 274 full views (67% more full views than abstract views). That article was only picked up by one media outlet. Overall, I found a r = -.49 (p = .01) correlation between the number of news-media pickups and the log of the ratio of full-text views to abstract views.

I suppose it's not surprising that articles picked up by the media attract more casual readers who will view the abstract only. I have no way of knowing that many of these readers are fellow academics in philosophy, psychology, and the other humanities and social sciences. But if so, and if my hypothesis is correct that academic researchers are increasingly exposed to psychological research based on Tweetability rather than methodological quality, that's bad news for the humanities and social sciences. Even if the rich really are jerks.


Rolf Degen said...

Thanks, Eric, you hit the nail on the head with that one: psychology is ruining its reputation with striving for sexy and tweetable results. That research about the rich being jerks has already been debunked on methodological grounds and because new research refuted it. Ever heard of it? Not sexy. No wonder that Diederik Stapel, the "King of Cheats",played that game like a master. His latest research, which never made it into a journal because he got busted, "proved" that vegetarians behave more ethically than meat eaters. The press release hit the headlines, and that study would have become THE hit of the year if he hadn't been caught. Look what he writes in his memoirs, it is an eye opener:

Eric Schwitzgebel said...

Yes, Rolf -- and thanks for your own work on these issues!

Callan S. said...

But what would this suggest - that academics quality of work before was founded on a lack of various forms of information? That they required being issolated to keep up 'quality'? That'd be fairly damning and indicative of an echo chamber being the prime constituant of quality, in such a case.

(though they might be found to have more criminal convictions or other "bad outcomes")

I think you'd have to look at charges laid rather than charges dropped. The rich have better lawyers, after all. Potentially making them look more moral.

chinaphil said...

I'm not too worried about academics. This is what universities are for: to teach the people who actually do this research what is valid and what isn't. If universities can't successfully train psychologists to apply filters (and supply institutional filters themselves), then it's the universities which need to step up their game.

The much bigger problem here is not the new one of social media, but the old one of interpreting a statistical average as an absolute result. Men are taller than women, but we're all familiar with the fact that sometimes a woman is taller than a man. But report that "the rich are jerks" or "northern Chinese are more individualistic" and lay readers will and do interpret it to mean that every rich person is a jerk and every northern Chinese is John Galt in waiting. Columnists will publish columns which literally say "I know a rich guy who's not a jerk, therefore the research is wrong".

Psychology is more vulnerable to this rampant misreading because its results are so salient and juicy. I think the answer is worldwide censorship of headlines, forcing all news sources to cast them in the form: "Science geeks claim: wealthier individuals may be greedier 60% of the time."

Anonymous said...

A couple of nitpicky points:

"Of course, another possibility is that the wealthy are just more willing to risk the cost of a parking ticket." Doesn't this still make them assholes though? The fact that they can afford to make that calculation doesn't really excuse the inconvenience this creates for other people. Of course your point on methodology still stands, I just thought it was an odd statement.

Re: the bias for or against the poor in studies, this strikes me as something that really depends on your political affiliation. Your news intake is heavily influence by your beliefs, so while a liberal may look for studies which "excuse" the poor, a conservative will often look for the opposite.

So you have to look at social media as something that is not only biased towards the sensational and pretty, but also something that's really balkanised and which aggregates stories based on their user's beliefs. And given that social media is really dependent on online advertising, I don't see how this can change for the better in the near future.

Anonymous said...

do you know as well?
I never wrote to you but you do philosophy of mind as well.
did they tell you?
I keep writing on philosophers anonymous.
so, please answer me.

Eric Schwitzgebel said...

Thanks for the comments, folks!

Callan: I'd say probably a mix of information from other sources and also less information in total about up-to-date research in psychology. The value or disvalue of having less information depends both on how trustworthy the information is and whether there are other sources of knowledge that would take its place.

chinaphil: I agree with your concern, but I disagree that we shouldn't be worried about academics! My view is that they are less different in their thinking from non-academics than we academics would like to believe.

Anon 4:20: Well, in that particular respect, perhaps. But there are other types of behavior that might go the other way given the economic incentives (such as soda-stealing). Perhaps the point was not as clearly made as it should have been. I entirely agree with your other points, though I do think that it's a fact about the current social structures in academia as a whole that it the group would be decidedly less keen on a study impugning the moral character of the poor than of the rich.

Anon 5:55: I'm not sure I understand what you're asking.

Anonymous said...

you do not read spiros blog?
I thought you do.

does chalmers tell you what he does to me?
do you know anything?
will you tell me the truth or will you lie to me?

Tamler said...

Hey Eric, great post, I agree with everything and I have no idea how it gets better. The only thing I'd add is the additional influence of "teachability". Sexier results are more fun to teach so the studies and our less than critical approach to them get passed on to new generations of students who continue the cycle. I've said on many occasions that I think trolley problem research is deeply flawed and on balance does more harm than good. Yet I still find myself teaching them and reporting results in intro classes! They always get a laugh, students perk up. And sexy results are far more likely to get popularized, so you have brilliant thinkers and minds like George Pelecanos (on a recent Fresh Air) regurgitating atrocious neuroscience research on adolescent brains. It's depressing.

Eric Schwitzgebel said...

Anon, I sometimes read Spiros, but I'm afraid I still don't know what you're talking about. Feel free to email me if it's something confidential.

Eric Schwitzgebel said...

Yes, Tamler -- that's an interesting point. Haidt's dumbfounding studies are also wonderfully teachable. Now, I'm not so down on trolleys or on dumbfounding, but your point leads me to wonder whether their teachability substantially raises their visibility and influence, even in the scholarly community itself.

Anonymous said...

I wrote you to this email address
the other one does not work for me