Rigor and Reproducibility
The recent evidence showing the irreproducibility of significant numbers of biomedical-research publications demands immediate and substantive action. The NIH is firmly committed to making systematic changes that should reduce the frequency and severity of this problem — but success will come only with the full engagement of the entire biomedical-research enterprise.
Collins, F., Tabak, L. Policy: NIH plans to enhance reproducibility. Nature 505, 612–613 (2014).
The NIH requires all research grants and RPPRs emphasize rigorous approaches to ensure robust and unbiased results. Each core at UNC is encouraged to develop Rigor and Reproducibility guidelines for their labs and to publicize these guidelines on their lab websites and make them available to core customers upon request.
What do Rigor, Reproducibility, and Transparency mean?
In 2015, the NIH developed guidelines around the need for clear standards and emphasis to enhance rigor and transparency in the studies it supported to drive reproducible results. This emphasis is twofold:
1) To establish and evaluate the rigor and reproducibility of past research which forms the bedrock for future research justifications and
2) to address how proposed research will achieve unbiased and robust results.
Scientific rigor is the strict application of the scientific method to ensure unbiased and well-controlled experimental design, methodology, analysis, interpretation and reporting of results. (Source: NIH Rigor and Reproducibility Webpage)
Reproducibility and Replicability are scientific terms that may have specific and distinct meanings within certain fields. One definition that the Office of Research Technologies uses to discuss reproducibility is:
- Related to repeatability and replicability. Refers to whether research findings occur. Broadly, reproducibility refers to direct replication, an attempt to replicate the original observation using the same methods of a previous investigation but collecting unique observations.Most broadly, reproducibility refers to conceptual replication, an attempt to validate the interpretation of the original observation by manipulating or measuring the same conceptual variables using different techniques. (Open Science Collaboration (p. 300-301))
Further elaboration on these terms can be found at The CURE
Transparency refers to the clarity and thoroughness of the methods and procedures used in an experiment that lends that experiment to reproducibility and replicability. It is often helpful to think of transparency in terms of writing a recipe. A transparent recipe would include all information required such that 10 different chefs could create 10 identical dishes.
For well-done examples of RRT statements, see the Microscopy Services Laboratory (Department of Pathology/LCCC; Pablo Ariel, Director) and the UNC Flow Cytometry Facility (LCCC; Ramiro Diz, Director)
Core facilities face unique challenges in supporting RRT initiatives. For tips and advice in navigating questions about RRT click here.
Guidelines for Rigor and Reproducibility from the NIH
Grant Application Instructions from the NIH
Guidelines for Reproducibility of Biophysics Research from the Biophysical Society
Standards for Reporting Enzymology Data (STRENDA) Guidelines from the Beilstein Institut
Guide to Rigor and Reproducibility for Flow Cytometry from the UNC Flow Cytometry Core
