Policy: NIH Plans to Enhance Reproducibility of Biomedical Research

In a recent editorial in the journal Nature, Francis Collins and Lawrence Tabak from the National Institutes of Health (NIH) discuss the concerns that NIH has with current lack of reproducibility in biomedical research, and propose steps for improving the repoducibility records. In their editorial they point out that we have traditionally considered science to be a self-correcting field. Given the expectation that over time, all reported works would be replicated by peers.

In the short term, however, this replication is not happening as oftern as it is necessary for science to maintain its public credibility. They state: "Preclinical research, especially work that uses animal models1, seems to be the area that is currently most susceptible to reproducibility issues. Many of these failures have simple and practical explanations: different animal strains, different lab environments or subtle changes in protocol. Some irreproducible reports are probably the result of coincidental findings that happen to reach statistical significance, coupled with publication bias."

In response to this concerns, "the NIH is developing a training module on enhancing reproducibility and transparency of research findings, with an emphasis on good experimental design. This will be incorporated into the mandatory training on responsible conduct of research for NIH intramural postdoctoral fellows later this year. "

More Data Transparency: "The NIH is also exploring ways to provide greater transparency of the data that are the basis of published manuscripts. As part of our Big Data initiative, the NIH has requested applications to develop a Data Discovery Index (DDI) to allow investigators to locate and access unpublished, primary data"

There is also an initiave for fostering open post-publication reviews. According to Collins and Tabak, "NIH launched an online forum called PubMed Commons (see go.nature.com/8m4pfp) for open discourse about published articles. Authors can join and rate or contribute comments, and the system is being evaluated and refined in the coming months. More than 2,000 authors have joined to date, contributing more than 700 comments."

Publishers are also dropping some of the artifial constrains (inherited from the era of physical paper publishing) that limits the description of methodology. Collins and Tabak point out that "Nature Publishing Group, the publishers of this journal, announced 8 in May 2013 the following: restrictions on the length of methods sections have been abolished to ensure the reporting of key methodological details"

And finally Collins and Tabak said that correcting the incentives for researchers is "Perhaps the most vexed issue is the academic incentive system." According to Collins and Tabak, "It currently over-emphasizes publishing in high-profile journals. No doubt worsened by current budgetary woes, this encourages rapid submission of research findings to the detriment of careful replication. To address this, the NIH is contemplating modifying the format of its 'biographical sketch' form, which grant applicants are required to complete, to emphasize the significance of advances resulting from work in which the applicant participated, and to delineate the part played by the applicant."

This article was first published in the Kitware Blog. The original post can be read here.