Are 15 UCLA Anthropology Graduate Students Representative of the Los Angeles Population? More Thoughts on Henrich's The WEIRDest People in the World

Earlier this month, I complained about Joseph Henrich's somewhat loose summaries of scientific research in his recent, influential book The WEIRDest People in the World.  At the time of the post, I had read through Chapter 6.

One of my complaints was that in explaining how research on economic games works, Henrich's paradigmatic fictional example described giving each research participant $20 to $30 per game over the course of ten games, totaling $200-$300 per participant.  However, economic games rarely have stakes that large.  More typically, the stakes are about a tenth of that.  Overstating the amount typically at stake illegitimately prevents the naive reader from forming the skeptical thought that people might behave differently with small amounts of laboratory money than with the larger amounts commonly at stake in real-world situations.

Despite my concerns, I find Henrich's book fascinating, and I am finding much of value in it.  So I kept reading.  Last week, I hit Chapter 9 and, given my complaints about Chapter 6, I was struck by the following paragraph:

These interviews contrasted with those I did in Los Angeles after administering an Ultimatum Game that put $160 on the line.  It was a sum that was calculated to match the Matsigenka stakes. [Matsigenka live in small farming hamlets in the Amazon.]  In this immense urban metropolis, people said they'd feel guilty if they gave less than half.  They conveyed the sense that offering half was the "right" thing to do in this situation.  The one person who made a low offer (25 percent) deliberated for a long time and was clearly worried about rejection.  

Wait, $160 per participant per game?!  

(In the Ultimatum Game, Person A is given a sum of money to split with Person B.  Person A proposes a split -- say, 50/50 or 80/20 -- and then Person B has the choice either to accept the resulting split or reject the offer, in which case neither player gets any money.)

I had to look up the study.  Indeed, Henrich did offer $160 to participants.  But -- understandably given the amounts at stake -- the sample size was very small: only 15 (that is, 15 people in the Person A role, whose offers provided the main data).  And those 15 people were all graduate students in the Anthropology Department at UCLA, paired with 15 other anthropology students.

While it's not exactly wrong for Henrich to summarize the data as he did, his presentation omits details that seem to me quite relevant and which might fuel a skeptical interpretation.  Should we consider 15 UCLA Anthro grad students representative of the Los Angeles population?  Henrich treats their behavior as representative without explicitly flagging for the reader how unusual a group they are.

In the original article, Henrich does make a case for choosing this population.  It's a group of acquaintances, like the Matsigenka population was a group of acquaintances.  Like the Matsigenka participants, the graduate students all personally knew the experimenter, Henrich himself.  That could potentially control for any inclination to be more generous in order to create a favorable impression on a high-status, high-resource acquaintance.  In the original article and in at least one later re-presentation of the work, Henrich explicitly acknowledges some of the potential concerns with taking these students as representative of the larger U.S. urban population.

But of course all of this is hidden beneath Henrich's description in his book of the participants as merely being from "the immense urban metropolis" of Los Angeles.  Given only that description of them, you might reasonably guess that the L.A. participants were strangers recruited off the streets.

Yes, readers can't be told every detail, especially in a book of such sweeping scope as Henrich's.  This creates a situation in which the reader must trust the author.  As an author, part of your job is to warrant that trust.  As a critical reader, part of your job is to assess as best you can whether the author in fact warrants trust.  One tool the reader can use spot checking, especially when the author enters areas where you have some independent sources of knowledge.

If you're inclined to trust Henrich's judgment that these 15 anthropology students were a well-chosen representative sample of Angelenos, then his omission is one you should feel comfortable enough with.  You should think, "I'm in good hands.  He's not distracting me with irrelevant details."  But my own sense is different.  Henrich omits crucial details about his population that I would want to know, and that I think readers in general should want to know so that they can think critically about the presented research.

-----------------------------------------------------------

By the way, Henrich replied on Facebook to my earlier blog post about the book.  If you're curious, check it out.  My sense is that his characterization of my post is inaccurate and that he did not correct that mischaracterization when given an opportunity to do so.  Please feel free to read my earlier post to judge whether I'm being fair in my complaint.

[image source]