- Open Access
Better reporting for better research: a checklist for reproducibility
© Kenall et al. 2015
- Published: 23 July 2015
How easy is it to reproduce or replicate the findings of a published paper? In 2013 one researcher, Phil Bourne, asked just this. How easy would it be to reproduce the results of a computational biology paper? . The answer: 280 hours. Such a number is surprising, given the theoretical reproducibility of computational research and given Bourne was attempting to reproduce work done in his own lab. Now at the National Institutes of Health (NIH) as Associate Director of Data Sciences, Bourne is concerned with the reproducibility of all NIH funded work, not just his own—and the problem is large. In addition to work in computational biology (which theoretically should be more easily reproducible than “wet lab” work), hallmark papers in cancer through to psychology have been flagged as largely unreproducible [2, 3]. Closer to home, GigaScience has carried out similar work to quantify reproducibility in their content. Despite being scrutinized and tested by seven referees, it still took about half a man-month worth of resources to reproduce the results reported in just one of the tables . “Reproducibility” is now increasingly on the radar of funders and is making its rounds in the wider media as well, with concerns of reproducibility making headlines at The Economist  and New York Times , amongst other outlets.
It is critical to note that irreproducible work doesn’t necessarily mean fraud occurred, nor even that the findings are incorrect; likewise, reproducible research can still be incorrect. While this key point is well-understood by most scientists, this is not always easy to explain to the general public. However, as most research is paid for through tax payers, public trust in research is essential. We—researchers, funders, and publishers—must do a better job at communicating this message to the public. We must better explain that science is an activity that continually builds on and verifies itself. But we also must develop policies that better support this process—policies, for example, that promote transparency and allow for improved verification of research.
Clearly important for clinical research, verification is equally important for preclinical research, something we all have an equal stake in. No one can innovate new drugs overnight, no matter how rich they are, no matter which doctor they see. Better, more robust preclinical research benefits us all.1 Our ability to rely on published data for potential therapeutics is critical, and recently its reliability has been called into question .
One well-publicised example of this was brought to light in an oncology study of preclinical research findings in which researchers were able to confirm only 11 % of the findings [8, 9]. Although the relevance of more robust research is clear in the area of oncology, it is also important for more exploratory research that might never make it to the preclinical setting. Funding and time are both increasingly limited, and the waste generated from follow-up work based on irreproducible research is high. A recent study by Freedman et al. estimated this at approximately $28 billion a year for preclinical research in the United States alone .
The NIH have recently taken bold steps to begin to tackle the need for better design, more appropriate analysis, and greater transparency in the conduct and reporting of research. In January 2014 the NIH announced they would fund more training for scientists in data management and restructure their grant review process to better value other research objects, such as data . But it is peer review and the editorial policies and practices of journals that have come under the greatest scrutiny, and in June 2014 a set of guidelines for reporting preclinical research were proposed by the NIH to meet the perceived need for more stringent standards . These guidelines ask journals to ensure, for example, that authors have included a minimum set of information on study design, that statistical checks have been carried out by reviewers, and that authors have been given enough information to enable animal strains, cell lines, reagents, and so on, to be uniquely identified reagents. (For a full list of requirements, see the NIH Principles and Guidelines for Reporting Preclinical Research.)
For further discussion of this around clinical trial transparency and reliability, see Ben Goldacre’s Bad Pharma.
To better support our authors in adhering to this checklist, we have also recently revised our section on data availability, detailing where authors can deposit their data and how to cite their data in their manuscript. We also have in-house staff available to work with authors to find a home for their data. http://www.biomedcentral.com/about/editorialpolicies#DataandMaterialRelease
The Center for Open Science with stakeholders from research have recently devised an easy to use set of guidelines based on eight standards and three levels of adherence. With this checklist, all journals will adhere to level 2 requirements. At present, all BioMed Central journals adhere to level 1 requirements. http://www.sciencemag.org/content/348/6242/1422.figures-only
We thankfully acknowledge the useful feedback on the checklist from Susanna Sansone at BioSharing (https://www.biosharing.org/), the entire BMC Biology and Genome Biology editorial teams, including Penny Austin and Rafal Marszalek, and the Research Integrity team, especially Maria Kowalczuk and Elizabeth Moylan (http://www.biomedcentral.com/authors/biomededitors), at BioMed Central. This editorial was published jointly in BMC Neuroscience, Genome Biology, and GigaScience.
- Garijo D, Kinnings S, Xie L, Xie L, Zhang Y, et al. Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome. PLoS ONE. 2013;8(11), e80278. doi:10.1371/journal.pone.0080278.View ArticlePubMedPubMed CentralGoogle Scholar
- Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Med. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124.View ArticlePubMedPubMed CentralGoogle Scholar
- Baker M. First results from psychology’s largest reproducibility test. Nature. 2015: http://www.nature.com/news/first-results-from-psychology-s-largest-reproducibility-test-1.17433
- González-Beltrán A, Li P, Zhao J, Avila-Garcia MS, Roos M, Thompson M, et al. From Peer-Reviewed to Peer-Reproduced in Scholarly Publishing: The Complementary Roles of Data Models and Workflows in Bioinformatics. PLoS ONE. 2015;10(7), e0127612. doi:10.1371/journal.pone.0127612.View ArticlePubMedPubMed CentralGoogle Scholar
- “Trouble at the lab” The Economist, Oct 19, 2013 http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble
- G Johnson. “New Truths That Only One Can See”, The New York Times, Jan 20, 2014 http://www.nytimes.com/2014/01/21/science/new-truths-that-only-one-can-see.html?_r=0
- Perrin S. Preclinical research: Make mouse studies work. Nature. 2014: http://www.nature.com/news/preclinical-research-make-mouse-studies-work-1.14913
- Ellis and Begley. Drug development: Raise standards for preclinical cancer research, Nature. 2012: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html
- G Kolata, “How a New Hope in Cancer Testing Fell Apart”, The New York Times, July 7, 2011 http://www.nytimes.com/2011/07/08/health/research/08genes.html?_r=1
- Freedman LP, Cockburn IM, Simcoe TS. The Economics of Reproducibility in Preclinical Research. PLoS Biol. 2015;13(6), e1002165. doi:10.1371/journal.pbio.1002165.View ArticlePubMedPubMed CentralGoogle Scholar
- Collins and Tabak. Policy: NIH plans to enhance reproducibility. Nature. 2014: http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586
- NIH Principles and Guidelines for Reporting Preclinical Research http://www.nih.gov/about/reporting-preclinical-research.htm
- Bustin S, Beaulieu J-F, et al. MIQE précis: Practical implementation of minimum standard guidelines for fluorescence-based quantitative real-time PCR experiments”. BMC Mol Biol. 2010;11:74. doi:10.1186/1471-2199-11-74.View ArticlePubMedPubMed CentralGoogle Scholar
- I Hrynaszkiewicz. “PRISMA Statement Published--and Endorsed by Biomed Central’s Journals”. 2009: http://blogs.biomedcentral.com/on-medicine/2009/07/27/prisma-statement-published-and-endorsed-by-biomed-centrals-journals/
- Please see the full checklist here (http://genomebiology.com/authors/instructions/minimum_standards_reporting). The BioMed Central Checklist can also be found in our collection on BioSharing (https://www.biosharing.org/collection/BMC).
- Center for Open Science, Transparency and Openness Promotion Guidelines https://osf.io/ud578/?_ga=1.173437419.933499240.1433864758
- Journals unite for reproducibility. Nature 515, 7 (06 November 2014) doi:10.1038/515007a
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.