How Often Are Philosophy Articles Actually Cited? Encouraging News

You hear terrible things.  You hear, for instance, that over 50% of philosophy articles are entirely uncited.  You hear that the average philosophy article is cited less than 5 times.  People will sometimes say things like the "median number of readers for a philosophy article is 1" or "there's not much difference between publishing a paper and throwing it away".  Both of the last two comments appear in a recent Twitter thread launched by Helen De Cruz.  It was reading this thread that inspired me to do the analyses I'll share today.

Generally my reaction to analyses and comments of this sort is to think that they are considerably underestimating how often philosophy articles are actually cited.  The "big data" interdisciplinary analyses often use methods that are a terrible fit for philosophy, such as looking at citations in the past two years.  (Often in philosophy it takes two years or more to write and publish an article.)  Also, I wonder what counts as an "article".  If we're including two-page book reviews, it's little wonder that they'd be little cited, and similarly if we're including publications in predatory or obscure journals.  What would be more interesting to know -- and what I think most people in this discussion really care about -- is how frequently cited are full-length research articles in "respectable" mainstream philosophy journals?

Before you look at my analyses, any guesses?

------------------------------------------------------------

Method: I selected six representative general philosophy journals for analysis: two "top ranked" journals (Philosophical Review and Nous), two mid-ranked journals (Canadian Journal of Philosophy and Pacific Philosophical Quarterly), and two unranked but reputable journals (Philosophia and Southern Journal of Philosophy).  I then downloaded the entire table of contents of these journals from the year 2012 -- giving a ten-year citation window -- and excluded anything that wasn't an ordinary full-length research article (e.g., book reviews, editors' introductions, symposium proceedings).  From each journal, I randomly selected 15 articles and noted their total number of citations in Google Scholar.

Although this is not a large database, the results from this carefully designed search are, I think, striking.

Overall, the mean article was cited 31 times, and the median number of citations was 20.  For the elite journals the mean was 50 and the median was 36; for the mid-ranked journals the mean was 25 and the median was 8; for the unranked journals, the mean was 17 and the median was 12.  Only three of the 90 articles (3%) were cited zero times.

Here's a more specific breakdown, journal by journal.  Of course with only 15 articles per journal, there will be lots of noise in the numbers when considered at this fine a grain.

[histogram of total citations by article for articles published in 2012 from six representative general philosophy journals; click to enlarge and clarify]

You might wonder if the articles that were cited just a few times were really just cases of self-citation (an author citing their own earlier work, rather than citation by another author); but this was not the case.  Although there was naturally a bit of self-citation, most citations were by other authors even for the articles with just a few citations.

You might wonder whether the articles mostly gather citations in the first several years, with citation rates falling off substantially by year ten.  I didn't collect these data systematically, but a sampling of articles finds generally that about half of the citations are since 2018, with no substantial decline at the end.  Presumably, then, these articles will continue to gather citations post-2022.  I would estimate that these articles have collected only half of the citations that they will eventually collect, and perhaps they will collect many more if there is a long temporal tail into the future.  If we double our citation estimates to account for this, the mean article will receive 62 citations over its lifetime and the median article will receive 40 citations.

ETA 12:37 p.m.: Since Google Scholar doesn't tend to include books as sources of citation (though it does include books as targets of citation), we should probably bump these numbers up even more.  Ballpark guess: between monographs and articles in anthologies, about one-third of all philosophy citations appear in books.  If that's right, actual citation rates would be about 50% higher.

I draw the following general conclusions:

(1.) Very few articles published in mainstream general philosophy journals remain entirely uncited.  My estimate of 3% might be off, due to the smallish sample size and the particular journals selected; but it's highly unlikely that non-citation rates of mainstream general philosophy articles are anything near 50%.

(2.) The majority of articles are cited often enough to be "in the conversation".  While an outsider to academia might not think that ten citations is much impact, I submit that a more appropriate perspective is this: If your article is cited at least ten times, then it is having an impact on other specialists in your subfield.  Your article isn't falling into the void.  It is part of the conversation, and other scholars are reacting to it.  A journal article needn't have a huge influence outside its subarea to be successful.  The beauty of academic journals is that they host technical pieces that often can only be appreciated by a few dozen specialists.  If academia is worthwhile, then continuing those specialists' conversations is worthwhile, and high citation rates for technical pieces should not be expected.  (I intend this ten citation criterion as an approximately sufficient condition for impact, not a necessary condition.)

(3.) Articles in elite journals are cited only about three times as often as articles in unranked but reputable journals.  To me, this was the most surprising result, and part of me wonders whether we'd see different results if we looked at different years or different unranked but reputable journals.  Philosophical Review is much more prestigious in the eyes of the typical mainstream academic philosopher than is the typical unranked but reputable journal.  I find this result encouraging, especially given the tiny acceptance rates at the elite journals (Philosophical Review reports accepting about twelve articles per year out of 600 submitted).  Publishing in unranked journals is not shouting into the void.  It is not defeat.  The dynamic is less jackpot-or-nothing than I would have expected.  You don't have to get into a top-ten journal to have an impact.  Philosophers do regularly read and cite articles from the less prestigious journals.

(4.) Citation skew is much less extreme than it could be.  Another respect in which philosophy citation practices are not jackpot-or-nothing is revealed by the smallish differences between the means and medians.  Of course the means are higher: There's right skew in the data, a tendency for high-end outliers to pull up the mean.  But it's not like most articles are cited 0-5 times and a few are a cited hundreds of times.  Authors can reasonably expect that a decent article in a decent journal will have at least a moderate impact on their subfield.