SCIENCE ADVANCES ON A FOUNDATION OF TRUSTED DISCOVERIES. REPRODUCING AN EXPERIMENT
is one important approach that scientists use to gain confidence in their conclusions.
Recently, the scientific community was shaken by reports that a troubling proportion of
peer-reviewed preclinical studies are not reproducible. Because confidence in results is of
paramount importance to the broad scientific community, we are announcing new initiatives
to increase confidence in the studies published in Science. For preclinical studies (one of the
targets of recent concern), we will be adopting recommendations of the U.S. National Institute of Neurological Disorders and Stroke (NINDS) for increasing transparency.* Authors
will indicate whether there was a pre-experimental plan for data handling (such as how to
deal with outliers), whether they conducted a sample size estimation to ensure a sufficient
signal-to-noise ratio, whether samples were treated randomly, and whether the experimenter
was blind to the conduct of the experiment. These criteria will be
included in our author guidelines.
There are a number of reasons why peer-reviewed preclinical
studies may not be reproducible. The system under investigation may
be more complex than previously thought, so that the experimenter
is not actually controlling all independent variables. Authors may not
have divulged all of the details of a complicated experiment, making
it irreproducible by another lab. It is also expected that through random chance, a certain number of studies will produce false positives.
If researchers are not alert to this possibility and have not set appropriately stringent significance tests for their results, the outcome is a
study with irreproducible results. Although there is always the possibility that an occasional study is fraudulent, the number of preclinical
studies that cannot be reproduced is inconsistent with the idea that all
irreproducibility results from misconduct in such research.
It is unlikely that the issues with irreproducibility are confined to preclinical studies
(social science has been equally noted, for example). Unfortunately, there are no equivalents
to the NINDS recommendations for other disciplines that provide a basis for requiring transparency across all fields. For the next 6 months, we will be asking reviewers and editors to
identify papers submitted to Science that demonstrate excellence in transparency and instill
confidence in the results. This will inform the next steps in implementing reproducibility
guidelines. Science Translational Medicine, a sister journal of Science, already enforces the
NINDS guidelines for preclinical studies. Both journals also are open to improving on the
NINDS recommendations for preclinical studies.
There is also a wide range of sophistication in the application of statistics displayed in
research analysis, ranging from practically no statistics, to the routine use of generic software packages, to the application of advanced methods that extract subtle signals from noise.
Because reviewers who are chosen for their expertise in the subject matter of a study may not
be authorities in statistics as well, statistical errors in manuscripts may slip through undetected. For that reason, with the advice of the American Statistical Association and others,
we are adding new members to our Board of Reviewing Editors from the statistics community to ensure that manuscripts receive appropriate scrutiny in their methods of data analysis.
Science’s standards have always been high, and these measures add to steps we have
already taken to increase transparency, such as requiring data accessibility. Nevertheless,
journals can only do so much to assure readers of the validity of the studies they publish.
The ultimate responsibility lies with authors to be completely open with their methods, all of
their findings, and the possible pitfalls that could invalidate their conclusions.
– Marcia McNutt
Marcia McNutt is Editor-in-Chief of Science.
www.sciencemag.org SCIENCE VOL 343 17 JANUARY 2014
*S. C. Landis et al., Nature 490, 187 (2012).