Thursday, October 20, 2022

How Often Are Philosophy Articles Actually Cited? Encouraging News

You hear terrible things.  You hear, for instance, that over 50% of philosophy articles are entirely uncited.  You hear that the average philosophy article is cited less than 5 times.  People will sometimes say things like the "median number of readers for a philosophy article is 1" or "there's not much difference between publishing a paper and throwing it away".  Both of the last two comments appear in a recent Twitter thread launched by Helen De Cruz.  It was reading this thread that inspired me to do the analyses I'll share today.

Generally my reaction to analyses and comments of this sort is to think that they are considerably underestimating how often philosophy articles are actually cited.  The "big data" interdisciplinary analyses often use methods that are a terrible fit for philosophy, such as looking at citations in the past two years.  (Often in philosophy it takes two years or more to write and publish an article.)  Also, I wonder what counts as an "article".  If we're including two-page book reviews, it's little wonder that they'd be little cited, and similarly if we're including publications in predatory or obscure journals.  What would be more interesting to know -- and what I think most people in this discussion really care about -- is how frequently cited are full-length research articles in "respectable" mainstream philosophy journals?

Before you look at my analyses, any guesses?


Method: I selected six representative general philosophy journals for analysis: two "top ranked" journals (Philosophical Review and Nous), two mid-ranked journals (Canadian Journal of Philosophy and Pacific Philosophical Quarterly), and two unranked but reputable journals (Philosophia and Southern Journal of Philosophy).  I then downloaded the entire table of contents of these journals from the year 2012 -- giving a ten-year citation window -- and excluded anything that wasn't an ordinary full-length research article (e.g., book reviews, editors' introductions, symposium proceedings).  From each journal, I randomly selected 15 articles and noted their total number of citations in Google Scholar.

Although this is not a large database, the results from this carefully designed search are, I think, striking.

Overall, the mean article was cited 31 times, and the median number of citations was 20.  For the elite journals the mean was 50 and the median was 36; for the mid-ranked journals the mean was 25 and the median was 8; for the unranked journals, the mean was 17 and the median was 12.  Only three of the 90 articles (3%) were cited zero times.

Here's a more specific breakdown, journal by journal.  Of course with only 15 articles per journal, there will be lots of noise in the numbers when considered at this fine a grain.

[histogram of total citations by article for articles published in 2012 from six representative general philosophy journals; click to enlarge and clarify]

You might wonder if the articles that were cited just a few times were really just cases of self-citation (an author citing their own earlier work, rather than citation by another author); but this was not the case.  Although there was naturally a bit of self-citation, most citations were by other authors even for the articles with just a few citations.

You might wonder whether the articles mostly gather citations in the first several years, with citation rates falling off substantially by year ten.  I didn't collect these data systematically, but a sampling of articles finds generally that about half of the citations are since 2018, with no substantial decline at the end.  Presumably, then, these articles will continue to gather citations post-2022.  I would estimate that these articles have collected only half of the citations that they will eventually collect, and perhaps they will collect many more if there is a long temporal tail into the future.  If we double our citation estimates to account for this, the mean article will receive 62 citations over its lifetime and the median article will receive 40 citations.

ETA 12:37 p.m.: Since Google Scholar doesn't tend to include books as sources of citation (though it does include books as targets of citation), we should probably bump these numbers up even more.  Ballpark guess: between monographs and articles in anthologies, about one-third of all philosophy citations appear in books.  If that's right, actual citation rates would be about 50% higher.

I draw the following general conclusions:

(1.) Very few articles published in mainstream general philosophy journals remain entirely uncited.  My estimate of 3% might be off, due to the smallish sample size and the particular journals selected; but it's highly unlikely that non-citation rates of mainstream general philosophy articles are anything near 50%.

(2.) The majority of articles are cited often enough to be "in the conversation".  While an outsider to academia might not think that ten citations is much impact, I submit that a more appropriate perspective is this: If your article is cited at least ten times, then it is having an impact on other specialists in your subfield.  Your article isn't falling into the void.  It is part of the conversation, and other scholars are reacting to it.  A journal article needn't have a huge influence outside its subarea to be successful.  The beauty of academic journals is that they host technical pieces that often can only be appreciated by a few dozen specialists.  If academia is worthwhile, then continuing those specialists' conversations is worthwhile, and high citation rates for technical pieces should not be expected.  (I intend this ten citation criterion as an approximately sufficient condition for impact, not a necessary condition.)

(3.) Articles in elite journals are cited only about three times as often as articles in unranked but reputable journals.  To me, this was the most surprising result, and part of me wonders whether we'd see different results if we looked at different years or different unranked but reputable journals.  Philosophical Review is much more prestigious in the eyes of the typical mainstream academic philosopher than is the typical unranked but reputable journal.  I find this result encouraging, especially given the tiny acceptance rates at the elite journals (Philosophical Review reports accepting about twelve articles per year out of 600 submitted).  Publishing in unranked journals is not shouting into the void.  It is not defeat.  The dynamic is less jackpot-or-nothing than I would have expected.  You don't have to get into a top-ten journal to have an impact.  Philosophers do regularly read and cite articles from the less prestigious journals.

(4.) Citation skew is much less extreme than it could be.  Another respect in which philosophy citation practices are not jackpot-or-nothing is revealed by the smallish differences between the means and medians.  Of course the means are higher: There's right skew in the data, a tendency for high-end outliers to pull up the mean.  But it's not like most articles are cited 0-5 times and a few are a cited hundreds of times.  Authors can reasonably expect that a decent article in a decent journal will have at least a moderate impact on their subfield.


Graham Harman said...

It's probably somewhat worse in continental philosophy, where articles are less emphasized than books, and where people tend to cite the big historical figures rather than each other.

Anyway, I enjoyed this analysis. Thanks for posting it.

Eric Schwitzgebel said...

Thanks, Graham! Yes, I'd guess that you're right. This analysis is definitely general mainstream Anglophone philosophy, and might not generalize to subdisciplinary journals, non-mainstream (including Continental), and non-Anglophone philosophy. (I have a particular sociological definition of "mainstream" in mind here, in terms of Leiter rankings and Stanford Encyclopedia citation.)

It would be lovely if there were some good citation databases that pulled citations from books. Actually, that reminds me to add that qualification to the post. I had meant to say it, but it slipped my mind in write-up this morning.

Arnold said...

Splintered Mind thru my 70s has been a guide in the evolution of philosophy's researching citing linking sensing feeling thinking...
...questioning philosophical changes of the last 10,000 years from theology to ontology for here on earth purposes...thanks...

Blogging philosophy can be a cutting edge into the future of...

VC Müller said...

Thanks for the work - and the encouragement.
Standard databases seem to say different things:
Perhaps there are obvious explanations, like what to count as 'citation', but I am not sure.

ajkreider said...

Just wondering out loud what this might look like for specialty journals. I could envision things going both ways, with some articles being cited more often (being more central to the subfield), and more never being cited (being too narrow for engagement).

Eric Schwitzgebel said...

Thanks for kind words, Arnold!

VC: Right. These science-based metrics are usually a bad fit for philosophy and often yield results that lack face validity among experts in philosophy. "Impact factor" is especially annoying!

AJ: My guess is that subfields with high uptake in high-citation fields like psychology and computer science will show higher citation rates, with more of a right skew for a small number of articles that get interdisciplinary uptake; and on the flip side history of philosophy will tend to have low citation rates.

Arnold said...

NPR cites philosophy articles...

William Shatner experienced profound grief in space. It was the 'overview effect'
October 23, 20227:00 AM ET, ENRIQUE RIVERA

swipe to search then click...