US grant pressure vs retractions: not a meaningful comparison

Image

A chart in The Scientist today (pictured, right) plots retractions against NIH grant success rates.

As DrugMonkey writes: ‘it appears truthy. Decreasing grant success is associated with increasing retraction rates. Makes a lot of sense. Desperate times drive the weak to desperate measures’.

Hmmm. Cut-throat competition to get grants a driver for sloppy, sensational or downright fraudulent work? Does make sense.

Problem is, it’s really hard to find evidence for this – and you definitely won’t find it in retractions. I think this is the wrong chart.  Prompted by @chemjobber, I have gathered data for what I think is the correct chart (pictured, below), which paints a different picture.

Image

Before I get to this chart, the thing to bear in mind about the recent rise in retraction notices, is that it’s likely a sign of raised awareness of levels of misconduct/sloppiness that were always there, but hidden. That is, the retraction notices are merely uncovering a little more of the tip of the iceberg. Evidence? Well, Daniele Fanelli has already ably demonstrated that some 1-2% of scientists admit misconduct in anonymous self-reported surveys. And the Journal of Cell Biology has been checking figures in submitted papers for a decade and its proportion of papers revoked because of manipulation has stayed constant at around 1% over that time. Yet retraction notices run below 0.05% of the literature even despite their ten-fold increase in recent years. Go figure. As Fanelli put it at the recent World Conference on Research Integrity in Montreal: ‘Retractions are the symptom of a growing solution – not a growing problem’.

Now about this chart. It’s a minor irritant that the number of retraction notices plotted are worldwide notices, not those from the US, yet they’re being compared to US funding success rates. What’s more problematic is that the author has plotted retraction notices, not retracted papers. Retraction notices often end up retracting work that was published many years before – say, for example, when NIH success rates were actually higher.

So I’ve replotted a chart using PubMed Reminer, to get retracted papers per year, then restricted them to just those with US addresses, or those with US govt PHS + NIH extramural funding.

Note of extreme caution: PubMed does not catch all retractions and I don’t know how good PubMed Reminer is. But, it’s as good as I was going to get on a quick trawl. Results are in this spreadsheet, and the picture is above.

I have grayed out the circles for 2011 and 2012 since the numbers are unfair: we will find many more retracted papers from that year in future retraction notices from 2013, 2014, 2015 and beyond.

You will see that there is little correlation between NIH success rates and retracted US publications or retracted papers with PHS support/NIH extramural support. There is a correlation with the total world’s retracted papers. You’ll also notice that the US share of retracted papers each year drops dramatically in recent years (but we already knew that – see these figures).

Now, as I’ve noted above, we probably won’t find out until, say, 2015, whether squeezed NIH success rates in 2011 and 2012 led to a higher number of retracted NIH-funded or US (or even world) papers. So maybe we’ll find out that it did. I’m not holding my breath.

The real conclusion: comparing retraction notices against NIH success rates isn’t much evidence for anything.

About these ads
This entry was posted in Uncategorized and tagged . Bookmark the permalink.

One Response to US grant pressure vs retractions: not a meaningful comparison

  1. ferniglab says:

    Great post. Drawing a straight line through an underpowered dataset isn’t gong to get us anywhere! i would point out that few journals carry out checks on image data like J Cell Biol, so a likely consequence of their stringency is that their retraction rate is likely to be lower than the average. After all, Turnitin has had a clear deterrent effect in reducing plagiarism by students, so a journal that publicly fights image manipulation is not going to be the first port of call for serial image manipulators.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s