Friday, April 30, 2021

Are 15 UCLA Anthropology Graduate Students Representative of the Los Angeles Population? More Thoughts on Henrich's The WEIRDest People in the World

Earlier this month, I complained about Joseph Henrich's somewhat loose summaries of scientific research in his recent, influential book The WEIRDest People in the WorldAt the time of the post, I had read through Chapter 6.

One of my complaints was that in explaining how research on economic games works, Henrich's paradigmatic fictional example described giving each research participant $20 to $30 per game over the course of ten games, totaling $200-$300 per participant.  However, economic games rarely have stakes that large.  More typically, the stakes are about a tenth of that.  Overstating the amount typically at stake illegitimately prevents the naive reader from forming the skeptical thought that people might behave differently with small amounts of laboratory money than with the larger amounts commonly at stake in real-world situations.

Despite my concerns, I find Henrich's book fascinating, and I am finding much of value in it.  So I kept reading.  Last week, I hit Chapter 9 and, given my complaints about Chapter 6, I was struck by the following paragraph:

These interviews contrasted with those I did in Los Angeles after administering an Ultimatum Game that put $160 on the line.  It was a sum that was calculated to match the Matsigenka stakes. [Matsigenka live in small farming hamlets in the Amazon.]  In this immense urban metropolis, people said they'd feel guilty if they gave less than half.  They conveyed the sense that offering half was the "right" thing to do in this situation.  The one person who made a low offer (25 percent) deliberated for a long time and was clearly worried about rejection.  

Wait, $160 per participant per game?!  

(In the Ultimatum Game, Person A is given a sum of money to split with Person B.  Person A proposes a split -- say, 50/50 or 80/20 -- and then Person B has the choice either to accept the resulting split or reject the offer, in which case neither player gets any money.)

I had to look up the study.  Indeed, Henrich did offer $160 to participants.  But -- understandably given the amounts at stake -- the sample size was very small: only 15 (that is, 15 people in the Person A role, whose offers provided the main data).  And those 15 people were all graduate students in the Anthropology Department at UCLA, paired with 15 other anthropology students.

While it's not exactly wrong for Henrich to summarize the data as he did, his presentation omits details that seem to me quite relevant and which might fuel a skeptical interpretation.  Should we consider 15 UCLA Anthro grad students representative of the Los Angeles population?  Henrich treats their behavior as representative without explicitly flagging for the reader how unusual a group they are.

In the original article, Henrich does make a case for choosing this population.  It's a group of acquaintances, like the Matsigenka population was a group of acquaintances.  Like the Matsigenka participants, the graduate students all personally knew the experimenter, Henrich himself.  That could potentially control for any inclination to be more generous in order to create a favorable impression on a high-status, high-resource acquaintance.  In the original article and in at least one later re-presentation of the work, Henrich explicitly acknowledges some of the potential concerns with taking these students as representative of the larger U.S. urban population.

But of course all of this is hidden beneath Henrich's description in his book of the participants as merely being from "the immense urban metropolis" of Los Angeles.  Given only that description of them, you might reasonably guess that the L.A. participants were strangers recruited off the streets.

Yes, readers can't be told every detail, especially in a book of such sweeping scope as Henrich's.  This creates a situation in which the reader must trust the author.  As an author, part of your job is to warrant that trust.  As a critical reader, part of your job is to assess as best you can whether the author in fact warrants trust.  One tool the reader can use spot checking, especially when the author enters areas where you have some independent sources of knowledge.

If you're inclined to trust Henrich's judgment that these 15 anthropology students were a well-chosen representative sample of Angelenos, then his omission is one you should feel comfortable enough with.  You should think, "I'm in good hands.  He's not distracting me with irrelevant details."  But my own sense is different.  Henrich omits crucial details about his population that I would want to know, and that I think readers in general should want to know so that they can think critically about the presented research.

-----------------------------------------------------------

By the way, Henrich replied on Facebook to my earlier blog post about the book.  If you're curious, check it out.  My sense is that his characterization of my post is inaccurate and that he did not correct that mischaracterization when given an opportunity to do so.  Please feel free to read my earlier post to judge whether I'm being fair in my complaint.

[image source]

8 comments:

Philosopher Eric said...

Let me get this straight. Henrich gifts various students that he personally knows with the potential make some good money by playing a game that seems to have no risk to them, and the results end up corroborating his thesis given what he found with tribal members of the Matsigenka that he also gifted this way? Is it really that bad professor? Did he have no concern about even the appearance of experiment tampering and cherry picking? Or was it a more direct “So apparently my position makes good sense”? How long will our soft mental and behavioral sciences keep things “business as usual”?

Somewhat related to this is a new book by Julia Galef which discusses a “scout” versus “soldier” mindset. This morning one of our friends did a post on it.

Jake said...
This comment has been removed by a blog administrator.
SelfAwarePatterns said...

Based on what you've shared, your criticisms seem fair to me. I know I've had the experience a few times where an author seemed to be getting things wrong in areas I was familiar with, and it did erode my trust for their assertions in other areas. It seems to me, when issues are pointed out, the best response is either to point out why they are in fact not issues, or thank the critic and take their points under advisement for future editions. Talking about what other reviewers are saying seems a bit ad hominem.

I'm also not a fan of how much gets socked away in endnotes. When reading a book, I feel obligated to read them, and they often contain gems, but they take us out of the narrative. And it reminds me of an old saying, "What the large print giveth, the small print taketh away." Often the endnotes introduce points that weaken or complicate the narrative. It seems like if the author feels it's important enough to warrant an endnote, maybe it should be worked into the actual narrative somewhere.

(Philosopher-Eric, thanks for the plug!)

Mike

Eric Schwitzgebel said...

Thanks for the continuing comments, folks!

Julia Galef's interviews are terrific! She brings in wonderful guests, and she is so good at asking probing questions. I've already bought and read The Scout Mindset, and I can recommend it too!

Phil Eric: It's not possible to know from the description whether the graduate students might have surmised the hypothesis in advance, but yes, that does seem like one of several possible worries about that population.

Mike: Yes, I do worry about that. My treatment of his endnote 32 in an endnote of my own was intentional. If an author is going to put crucial qualifications in endnotes, then I can put qualifications of my objections to that author's view, based on the qualifications expressed those endnotes, in my own endnotes, right? Seems fair!

Eric Schwitzgebel said...

Jake, thanks for the compliment, but it was vague enough to raise concerns that you are a bot. Confirm your humanity and repost if you like!

Jake said...

Hahaha, sorry I failed the Turing test. I didn't have anything substantive to say about Henrich's book, but I have been reading your blog a lot recently, so I wanted to say something nice on a recent post! One of my favorite entries is your post about taking life advice from people on their deathbeds.

Philosopher Eric said...

Right professor, the students probably didn’t know the hypothesis. But if let out, that also might have gotten messy. Hopefully graduate students would know that they shouldn’t know the point of the game. Conversely if Henrich merely knew these students then he might also have had a reasonable sense of how they’d play the game. Why jeopardize his thesis by picking people that he didn’t know and thus might play the game in ways that contradicted his thesis?

I hope that you continue on with his book however. I consider such demonstrations very important for the future health of our still very troubled mental and behavioral sciences.

Jake,
Well done. I figured that clicking your name would lead me somewhere to buy something. If you’re a bot however, I’m sold!

Eric Schwitzgebel said...

Thanks for the kind words, Jake and Phil E!