23 MAY 2014 • VOL 344 ISSUE 6186 789 SCIENCE sciencemag.org
guest-edited by Nosek and Lakens, is less
reassuring. All told, the researchers failed
to confirm the results of 10 well-known
studies, such as the social psychological effects of washing one’s hands, holding cups
of warm or cold liquid, or writing down
flattering things about oneself. In another
five cases, the replications found a smaller
effect than the original study did or encountered statistical complications it did
not report. For embodied cognition and
also for behavior priming—the study of
how exposure to one stimulus, such as the
word “dog,” changes one’s reaction to another, such as a photo of a cat—the results
are particularly grim. Seven of the replications focused on experiments in these areas, and all but one failed.
No one is suggesting misconduct in any
of the original studies, but the results are
further blows to a field shaken several years
ago when a towering figure in priming research, Diederik Stapel, confessed to faking
data (Science, 7 December 2012, p. 1270).
And earlier this month, Jens Förster of
the University of Amsterdam, a pioneer of
embodied cognition research, was accused
by a Dutch government-appointed ethics
panel of data manipulation—charges he denies (Science, 9 May, p. 566).
Nor should the results be taken as a general indictment of psychological research,
because the targeted studies were not a
random sample, Nosek says. “They are entirely cherry-picked,” he says, based on the
importance of the original study and the
feasibility of replicating it.
Some of the authors of the targeted studies, however, feel not just singled out but
persecuted. Schnall, for example, contends
that the replications were not held to the
same peer-review standard as her original
studies. “I stand by my methods and my
findings and have nothing to hide,” she says.
The replications did employ an alternative model of peer review, called preregistration, promoted by the Center for
Open Science, a nonprofit organization co-founded by Nosek (Science, 30 March 2012,
p. 1558). Before any data were collected, the
replicators submitted their experimental
design and data analysis plan to external
peer reviewers, including the principal investigator of the original study. The subsequent data analysis and conclusions were
reviewed only by Nosek or Lakens.
Schnall contends that Donnellan’s effort
was flawed by a “ceiling effect” that, es-
sentially, discounted subjects’ most severe
moral sentiments. “We tried a number of
strategies to deal with her ceiling effect
concern,” Donnellan counters, “but it did
not change the conclusions.” Donnellan
and his supporters say that Schnall simply
tested too few people to avoid a false posi-
tive result. (A colleague of Schnall’s, Oliver
Genschow, a psychologist at Ghent Univer-
sity in Belgium, told Science in an e-mail
that he has successfully replicated Schnall’s
study and plans to publish it.)
Some replicators leaked news of their
findings online, long before publication
and in dismissive terms. On his personal
blog, Donnellan described his effort to re-
peat Schnall’s research as an “epic fail” in a
December post titled “Go Big or Go Home,”
which was then widely circulated on Twit-
ter. Donnellan defends the early announce-
ment. “I feel badly, but the results are the
results,” he says.
Schnall, however, says that her work was
“defamed.” She believes she was denied a
large grant in part because of suspicions
about her work and says that a reviewer of
one of her recently submitted papers “raised
the issue about a ‘failed’ replication.” She
adds that her graduate students “are wor-
ried about publishing their work out of fear
that data detectives might come after them
and try to find something wrong.”
Other researchers whose work was tar-
geted and failed to replicate told Science
that they have had experiences similar to
Schnall’s. They all requested anonymity, for
fear of what some in the field are calling
“replication bullying.”
Yet some whose findings did not hold up
are putting a positive spin on the experi-
ence. “This was certainly disappointing at a
personal level,” says Eugene Caruso, a psy-
chologist at the University of Chicago Booth
School of Business in Illinois, who in 2013
reported a priming effect—exposing people
to the sight of money made them more ac-
cepting of societal norms—that failed to
replicate. “But when I take a broader per-
spective, it’s apparent that we can always
learn something from a carefully designed
and executed study.” Caruso now has a
larger and more nuanced version of his
study under way.
The replications in psychology reflect a
growing trend in science (see table). The
field’s bruising experience shows that such
efforts should be handled carefully, stresses
Daniel Kahneman, a psychologist at Princ-
eton University, whose work was success-
fully replicated by the Many Labs team.
“The relationship between authors and
skeptics who doubt their findings is bound
to be fraught,” he says. “It can be managed
professionally if the rules that apply to both
sides are clearly laid out.”
To reduce professional damage,
Kahneman calls for a “replication eti-
quette,” which he describes in a commen-
tary published with the replications in
Social Psychology. For example, he says,
“the original authors of papers should be
actively involved in replication efforts”
and “a demonstrable good-faith effort to
achieve the collaboration of the original
authors should be a requirement for pub-
lishing replications.” In the case of this
week’s replications, “the consultations did
not reach the level of author involvement
that I recommend.” However, he notes that
“authors of low-powered studies with spec-
tacular effects should not wait for hostile
replications: They should get in front of the
problem by replicating their own work.”
For his part, Nosek hopes that the ten-
sions will be short-lived growing pains
as psychology adjusts to a demand, from
within and outside the field, for greater ac-
countability. “Our primary aim is to make
replication entirely ordinary,” he says, “and
move it from a threat to a compliment.” ■
Repeat after me
Select efforts in replication of research
EFFOR T REPLICATION TARGE T
Reproducibility Project: 50 high-impact cancer studies published from 2010 to 2012
Cancer Biology
Reproducibility Project: Articles published in 2008 from three psychology journals
Psychology
Reproducibility Initiative Hub for authors to request independent
replications of their experiments
Many Labs project Global network for orchestrating large replications
Reproducibility in Checks software code in 613 applied computer science papers
Computer Science
Crowdstorming project More than 50 analysts address same research
question using shared data set
