What makes a journal important?

The scientific community values citations, and with good reason. But, in other publishing arenas, the evaluation is based on how often the publication is read, not how often it is cited. This is true in traditional publishing with best-seller lists, and also in modern web publishing with metrics for visits and click-throughs. So, I was curious what relationship exists between how often works from a journal are cited, and how often articles from those journals are read by our users. Read on to find out:

A few years ago, I wrote a set of python scripts to improve on the services offered by PubCrawler and MyNCBI. Using some enrichment statistics and the Entrez API, I began emailing myself article alerts once a week without the false positive results inherent when searching PubMed for an author name or keyword. And when I first considered expanding my little set of python scripts into what is now PubChase, I began collecting a little polling data. At the department social hour, or at dinner with a seminar speaker, I asked how people found new literature to read. Answers involved RSS feeds and various heuristics for PubMed searches, journal club discussions, and dedicated blog reading. I suspect these days it would involve a fair amount of Twitter-gazing. But the most common answer in these informal surveys was to simply read the table of contents of a few journals each week. To me, the obvious follow-up question was how one should select which few journals of the thousands available to consider. In other words: what makes a journal important?

Now, it’s no mystery which three journals were invariantly listed in the response. After all, there is a pervasive (if dubious) belief that the awards of fellowships, faculty positions, and tenure are all predicated on “SNC” publications¹. And in fairness, a lot of folks told me that they also took time to look in one or two other journals, typically specific to their research interests or a “second tier” journal. But the lesson was that a frightening proportion of scientists were limiting themselves to Science, Nature, and Cell, maybe with a smattering of Genetics, Neuron, NSMB, etc.

And it’s also no mystery why these are the journals exalted to the lofty status of “worth-one’s-time-to-read-stuff-in-them.” Articles in these journals are, on average,² cited more often, and accordingly their Impact Factor is high. Citation count and Impact Factor are really easy statistics to understand: journals with high ones are the “winners.” But, if Impact Factor is the all-mighty reflection of journal quality,³ then why is it that Science, Nature, and Cell are not the three top journals by this metric?

Any good graduate student knows the answer to this question too: the other journals with high impact factors can be dismissed for targeting a clinical audience, for being a “review” journal, or for being a physics journal that got lost in the heavily biology-biased list of “science” journals. Because clinicians don’t win Nobel prizes too much these days. And reviews are only c.v. fodder with no value to the scientific community. And physics has too much math. So, after these dismissals, Nature is indeed the top-ranked journal, with Cell and Science close behind (after stumbling through crags of Nature specialty journals). Voila. Justification attained. We can all resume a frenetic SNC-or-Bust pursuit of scientific external validation.

But, before we do, I thought I’d share with you a few observations from the data comprised by our PubChase user base and the articles they chose to read.

The PubChase algorithm uses a “Bayesian-ish” model to predict which articles will be of interest before they accrue citations. This prognostication is required if you, like me, don’t want to wait around for a year or two to find out what to read. And it considers, basically, things that make sense to consider. These considerations include an estimate of the value of each journal, but unlike Impact Factor, which is based on citations, our model is based on who is reading the articles in a particular journal. And when I take a look at our internal rankings of journals in the model, a couple of things really stand out: 1) Science, Nature, and Cell are not the top 3 journals, and 2) maybe we should hit the breaks on dismissing all those reviews.

Here are, as of today, the top 20 journals according to our model:

1 Nature reviews. Genetics
2 Genetics
3 Genome biology
4 Genome research
5 PLoS biology
6 Genes & development
7 Cell
8 Molecular biology and evolution
9 Nature methods
10 Nature reviews. Molecular cell biology
11 Trends in genetics : TIG
12 Developmental biology
13 Journal of molecular biology
14 Science (New York, N.Y.)
15 Yeast (Chichester, England)
16 Development (Cambridge, England)
17 Current opinion in genetics & development
18 Molecular cell
19 Nature
20 Molecular systems biology

So, first things first: Bravo Cell! You made the top 10, unlike your SNC brethren.

Now, I would like to express this caveat: it’s clear to me from many observations that our users are enriched for the more quantitative sub-disciplines of biology, most notably genomics and population genetics. So, this likely explains the heavy representation of journals starting with the letters “G-E-N” in the list.

Interestingly, PLoS Biology takes one of the top spots, which agrees with an undercurrent of my informal polling suggesting that PLoS was on the cusp of joining the vaunted SNC trinity ^">4.

You can also see that developmental genetics is alive and well as a field of research⁵.

And it’s also quite clear that those c.v. cluttering review articles might actually have some value. I asked how many of the over 22,000,000 articles available in PubMed are from these journals. The answer is 0.92%.⁶ Then, I asked what percentage of articles read by our users were from these journals. The answer is 4.18%. If you want to know if our users are reading reviews more often than they would be by chance, then you could compare the last two percentages⁷. And with this many events, it’s not surprising that a difference of roughly 1% and 4% is super-significant.⁸ So, it turns out that while major advances in science are not published first in review form, a lot of the articles read are reviews. This gets to a bigger question perhaps best saved for conversational pub fodder regarding exactly to whom rewards are given for what in academia, what the roles of discovery versus communication should be, and so forth. But to me at least, this is a clear rejection of the idea writing a review is a valueless endeavor. Lots of people are reading reviews.

So, one last thought: I compared our rankings to journal Impact Factors for last year. I expected some relationship between citation number, which principally drives Impact Factor, and the PubChase score, but I was not sure how strong this relationship would be. In essence, I’m asking what is the relationship between citation events and reading events. Again a caveat: keep in mind this is just a scatter plot of 20 data points, selected from the tail of a distribution. The correlation coefficient for this plot is about 0.2.⁹¹⁰ It could be that the PubChase user base is not representative of the biological sciences as a whole. Or, it could be that the relationship between what is read and what is cited is not particularly strong. Or most likely, these data are structured with respect to multiple factors. Nonetheless, the relationship between readership and Impact Factor is mild at best.

Feedback and commentary are certainly welcome.

1. I was a little horrified, too, to see that some very accomplished researchers actually split out these publications on their lab websites (e.g. this one). It makes me wonder, as a reader, what such authors think of the articles they have published in other journals. [↩]
2. I’m actually quite mystified why Impact Factor, and basically all other metrics of journal quality continue to use only the mean citations as the only estimator of quality. It seems obvious to me that each journal has a true- and false-positive rate for generating truly high-impact work, and one could, given access to the sort of data that Google Scholar and ISI has amassed, easily explore the other distribution characteristics of citations per article for each journal. It seems intuitively true that the citation counts of articles in any journal are distributed exponentially, thus assuming that the majority of articles are as good as the article of mean quality in a given journal is super duper wrong. I suspect it’s true that in journals of all tiers, there are a minority of good articles that are allowing the remainder of the articles to hitchhike on the backs of the reputations earned by the minority of articles with good citation numbers. [↩]
3. Even ISI doesn’t think this is true, by the way. In this article from 1994, they explicitly say as much: “Thomson Reuters does not depend on the impact factor alone in assessing the usefulness of a journal, and neither should anyone else.” [↩]
4. <head spins while contemplating the nested acronym possibilities> [↩]
5. Though I’ve heard it suggested recently that reading G&D was tres passé. [↩]
6. I took a quick proxy for how many journals were dedicated to reviews just by asking what percentage contained the word “review” in their title I know this isn’t perfect, but it’s almost certainly an underestimate. The answer is 4.55%. [↩]
7. Or, rather, their proportions. [↩]
8. a Fisher’s Exact p-value on these proportions is very closely approximated by the number zero [↩]
9. I’d be happy to recreate this graph for thousands of journals, if I could get my hands on the full list of Impact Factors, or better yet, Google Scholar citation counts [↩]
10. Which I tried to do with some foul-minded html scraping, but quickly learned that Google isn’t keen on me scraping Scholar results. [↩]