Lida Anestidou, National Academy of Sciences
Steve Niemi, Liaison to the American College of Laboratory Animal Medicine
Restoring Faith in the Research Enterprise: A Call to Action
Malcolm Macleod, University of Edinburgh
Henry Bourne, University of California San Francisco
Evidence indicates that potential causes of scientific irreproducibility are multi-factorial, including insufficient reporting of details pertaining to study design and planning; inappropriate interpretation of results; and author, reviewer, and editor-abstracted reporting, assessing and accepting studies for publication. This panel presented and discussed the historical context of this problem, concerns and lessons from the Cochrane collaboration that may apply to research that uses animals as surrogates for humans and other animals, and systemic issues in the United States’ scientific research enterprise that precipitate the methodological problems leading to irreproducible research.
Citizens and Science: How Reproducibility Directly Impacts Public Perceptions
Robert Bazell, Yale University
Jan Piotrowski, The Economist
The panelists engaged the audience in a broader conversation about the importance of the public’s opinions to researchers and the extent to which these should be taken into consideration; why the public cares about using animals in research responsibly; what matters to the public about animal use for research; and credible sources of publicly available information on the use of animals in research, and making sure the information is accurate.
Great Expectations – Critical Assessment of Published Research: A “Mind’s-On” Exercise for Workshop Attendees
C. Glenn Begley, TetraLogic Pharmaceuticals
This is an audience-participation exercise designed to engage attendees, encourage learning and understanding of the issue of reproducibility and inspire a commitment to addressing and overcoming the challenges of reproducibility to safeguard the integrity of science. In this session, workshop attendees were presented with a collection of exemplars reflecting key elements of scientific publications linked to reproducibility techniques.
Heard but Not Learned? Impact and Outcomes of Previous ILAR Efforts
Jeffrey Everitt, GlaxoSmithKline
Coenraad F. M. Hendriksen, Netherlands Vaccine Institute
The issue of reproducibility in science is not new. It has been recognized as a pernicious problem in biomedical research worldwide for more than a decade. ILAR has invested significantly in addressing the issue, first in a workshop highlighting the special challenges of animal research in a global environment (Coenraad Hendriksen was chair, and more recently in assembling a subcommittee to produce guidelines for scientific publications involving animal studies (Jeffrey Everitt was chair). The impact of these efforts, however, has not met expectations, as concerns about reproducibility have grown rather than lessened. This session will put today’s workshop in context with the previous ILAR efforts to address the topic, objectively recounting the intent of those previous efforts, where they succeeded and where they fell short, and how we can learn from the past to ensure maximum effectiveness and measurable outcomes.
In addition to efforts by ILAR, a number of other organizations and entities have taken numerous approaches to address the issue of reproducibility in scientific research. This session is devoted to presenting a summary review of these actions to date, describing the “boundaries” and reach of their intended effects, account for how these actions have impacted reproducibility in science, and indicate what factors are not addressed by these actions and thus still fail to reduce the reproducibility problem.
Russell and Burch Revisited: Reconciling “Reproducibility” with “Replacement, Reduction, and Refinement”
Michael Festing, Independent Consultant
Stephen Latham, Yale University
There are growing calls by any number of groups to reduce, if not eliminate the use of most, if not all, animals in research. The scientific community’s response has been to “replace, reduce and refine” the use of animals in scientific studies wherever possible. However, has the earnest effort in addressing the “3R’s” actually contributed to the issue of reproducibility in scientific studies? Has, for example, the goal to reduce the number of mice to minimum necessary to achieve statistical significance actually left experiments with insufficient numbers per treatment group for reproducibility?
In the context of declining budgets, reduced support for basic research, lab closures and layoffs, and extensive specialized training with no guarantees for faculty positions, research scientists spend most of their time writing grant applications, knowing full well that only a few will earn a fundable score, and even fewer will win an award. Add this to the stress to publish research findings often and increasing teaching and service responsibilities, lest one fails to achieve tenure or keep the laboratory doors open … and even then, with tightening university budgets, there is no guarantee of career survival. This session explored the potential conflict between the impact of economics on science and the value of scientifically rigorous research as (dis)incentives of engaging in best practices.
Reproducibility Challenges in the Future of Animal Models
Roger Reeves, Johns Hopkins University
Jeffrey Rogers, Baylor College of Medicine
Monte Westerfield, University of Oregon
Rapid and dramatic advances in scientific technologies are driving a profound paradigm shift in biomedical research, especially in studies relying on the use of animal models. For example, in the past, human disease research in animals was often based on testing possible mechanisms of disease as represented in a population. By comparison, advanced technologies now allow us to use genomic, metabolomics, and proteomic data from a single human patient to inform the derivation of animal models that reflect the precise molecular mechanisms underlying an individual pattern of disease. Can such experimental models eliminate the need for blindedness or randomization? This session will explore the special considerations next-generation animal models pose to ensuring experimental reproducibility.
To fully address the challenges and issues associated with reproducibility in science, a holistic and comprehensive assessment of traditional methods for publicizing research results is needed. This session intends to posit creative and innovative ideas that seek to address the limitations of current practices faced by authors and publishers that neither reduce irreproducibility nor promote reproducibility. For example, negative experimental findings that result from sound scientific methodology can be as, if not more, informative than positive findings, yet these negative results often have no venue for public presentation in journals. Could journals, for example, consider pre-publication approval of a study submitted by a prospective author as a means to promote best scientific practices prior to beginning experiments? These and other provocative ideas are intended to emphasize the reproducibility of scientific experimentation early on during experimental planning and design, rather than later on, trying to recover a publication from a poorly designed study.
IOs, Vets, and IACUCs – Making Internal Regulators Partners in Reform
Kathryn Bayne, Association for Assessment and Accreditation of Laboratory Animal Care (AAALAC) International
Stuart Zola, Emory University
Jerry Collins, Yale University
Research scientists don’t operate in a vacuum or exist on an island. While not necessarily directly involved, the institutional environment, administration, operating procedures, and resource and facility infrastructure can significantly influence the planning, conduct, and interpretation of research studies. Such resources can thus affect the reproducibility of the research that utilizes them. However, scientists themselves will have little if any role to mitigate the effects these resources might have on a study’s reproducibility. This session will propose means and processes by which institutional officials and other persons involved in the administration can partner with researchers to implement best practices that reform the conduct and reporting of experimental studies and enhance reproducibility.
An Ounce of Prevention is Worth a Billion of Cure: Proactive Planning in the Preclinical Research Arena
John P. A. Ioannidis, Stanford Prevention Research Center
Paul Braunschweiger, CITI Program
Ghislaine Poirier, GlaxoSmithKline
This workshop has focused on problems in experimental reproducibility, how to recognize these problems, and what scientists and administrators can do to alleviate these problems. However, beyond ensuring that any study results can be replicated and reproduced by an objective third party, what about the impact of those study results, especially on creating new knowledge that contributes useful information in the preclinical arena? This session will discuss mechanisms by which the academic and commercial sectors can build bridges of communication early on in the pre-competitive space to further enhance the reproducibility of animal-based studies. It will also present some potential solutions. such interactions have the additional potential to catalyze and accelerate innovative discoveries that can have a transformative impact on science and human health.
Kent Lloyd, University of California, Davis