Originally published in December 2014 and comprehensively updated/edited in July 2026.
How failed replications in psychology and biomedicine sparked a reproducibility crisis, and the reforms—preregistration, open data—built to fix it.
By the early 2010s, a strange kind of anxiety had settled over the social and biomedical sciences — not the usual worry about funding or racing to publish first, but something closer to an identity crisis. Researchers were beginning to ask a question that should have been settled long ago: if we ran this experiment again, would we get the same result?
For most of the twentieth century, that question sat quietly in the background of scientific practice, assumed rather than tested. Replication — repeating a study to see if its findings hold up — was supposed to be baked into the process, a kind of built-in quality control. But by 2014, a slow accumulation of uncomfortable evidence was making it clear that a surprising number of published findings, some of them famous, some of them foundational to entire subfields, might not hold up at all. Journalists started using the phrase “replication crisis.” Some scientists began describing the scale of the problem in increasingly urgent terms. And a field that had spent decades building public trust in its authority suddenly found itself explaining, in fairly plain terms, why so much of what it had published couldn’t be reproduced.
This mattered for reasons that went well beyond academic pride. Psychology findings shape parenting advice, classroom design, and corporate training programs. Biomedical findings shape which drugs get tested in humans and which treatments doctors recommend. When the foundation under those findings turned out to be less stable than assumed, the consequences reached far past university walls. The reproducibility challenge, as it came to be called, wasn’t just a story about statistics. It was a story about incentives, about the psychology of scientists themselves, and about what happens when an entire professional culture optimizes for something other than getting the right answer.
A Discipline Starts Questioning Itself
Two events in 2011 are often pointed to as the moments the conversation shifted from a quiet, technical concern shared among methodologists into something the wider scientific community could no longer ignore.
The first was the publication of a study by Cornell psychologist Daryl Bem in one of psychology’s most respected journals, the Journal of Personality and Social Psychology. Bem’s paper reported experimental evidence for precognition — the idea that people could sense future events before they happened, including future random events that hadn’t occurred yet. The studies followed standard psychological methods, used conventional statistical tests, and passed peer review at a top-tier outlet. And yet the conclusion asked readers to accept that time could, in some sense, run backward.
The paper produced widespread disbelief, along with intense methodological scrutiny. That scrutiny didn’t stay focused on parapsychology for long. Critics began pointing out that if a paper using entirely conventional, “properly done” methods could produce evidence for something that almost certainly wasn’t real, then those same methods might be quietly producing false positives across the rest of the field, just for less obviously implausible claims. The Bem episode became a kind of stress test that the discipline’s ordinary statistical practices had failed.
The second event was less abstract and considerably more damaging to individual trust. Later that year, Tilburg University in the Netherlands suspended a prominent social psychologist named Diederik Stapel after colleagues raised concerns about his data. An investigation eventually determined that Stapel had fabricated data outright, inventing results for dozens of published papers and several doctoral theses supervised under his name, often without ever running the experiments he claimed to have conducted. By the time the dust settled, dozens of his papers had been retracted.
Stapel’s case was fraud, not a statistical or methodological failure, and it’s worth keeping that distinction clear. Most of the reproducibility problem has nothing to do with dishonesty. But the scandal did something important psychologically: it forced the field to ask how a fabrication of that scale could go undetected for over a decade, passing through peer review, coauthorship, and academic promotion committees along the way. If outright invented data could slip through unnoticed, what did that say about the more ordinary, unintentional distortions that might be happening everywhere else?
Putting Numbers to the Problem
Concern about reproducibility didn’t begin in 2011. Years earlier, in 2005, epidemiologist John Ioannidis published an essay in PLOS Medicine with a title designed to get attention: “Why Most Published Research Findings Are False.” Ioannidis wasn’t accusing scientists of fraud. He was making a mathematical argument. When a research field studies many possible relationships, most of which turn out not to be real, and when studies tend to be small and statistical thresholds are lenient, then a large share of the “significant” results that get published are likely to be false alarms rather than genuine discoveries. Smaller sample sizes, more flexibility in how a study is designed and analyzed, and stronger financial or career incentives to find something exciting all make the problem worse.
Ioannidis’s argument was theoretical, built from probability rather than direct observation, and it drew debate over exactly how pessimistic the numbers should be taken. But it gave the field vocabulary and a framework that later, more empirical projects would go on to test directly.
One of the most influential of those empirical tests came from psychology itself. In 2015, a large international collaboration known as the Reproducibility Project: Psychology, coordinated by the Center for Open Science and involving more than two hundred researchers, published the results of an ambitious effort: attempting to replicate 100 studies originally published in three major psychology journals. Using the original materials wherever possible and larger, more statistically robust samples, the team found that only around 36 to 39 of the 100 replications produced a statistically significant effect in the same direction as the original study. Even among the replications that did “succeed,” the size of the effect was typically about half of what the original paper had reported.
The biomedical sciences turned out to have a version of the same problem, arguably with higher stakes. In 2012, C. Glenn Begley, a former head of cancer research at the biotechnology company Amgen, and Lee Ellis of the MD Anderson Cancer Center published a commentary in Nature describing an internal effort at Amgen to confirm 53 “landmark” preclinical cancer studies before building further drug development work on top of them. Using teams of scientists working for up to a year on each project, Amgen’s researchers were able to reproduce the original findings in only a small fraction of cases — by some accounts as few as six of the fifty-three. A separate, similar effort at the pharmaceutical company Bayer around the same period found comparable results in its own cancer research portfolio, reproducing findings in less than a quarter of the projects it examined. These weren’t obscure papers. Many had come from well-regarded labs and had been published in prestigious journals, which was precisely what made the failures so unsettling: prestige and rigor, it turned out, weren’t the same thing.
It’s worth pausing on what a failed replication does and doesn’t mean. It doesn’t automatically mean the original researchers did anything wrong, let alone that they lied. Small differences in populations studied, in the exact procedures used, in the timing or setting of an experiment, can genuinely produce different results even when everyone involved acted in good faith. Some findings may be real but smaller or more context-dependent than first reported. Still, taken together, these projects made an uncomfortable pattern hard to dismiss: across disciplines as different as social psychology and cancer biology, a substantial share of published, peer-reviewed findings were failing to hold up under closer scrutiny.
The Hidden Flexibility Behind a “Significant” Result
To understand why this kept happening, it helps to look past headline statistics and into the day-to-day decisions researchers make while designing and analyzing a study. A body of methodological work from this period, most notably a widely cited 2011 paper by psychologists Joseph Simmons, Leif Nelson, and Uri Simonsohn, drew attention to something they called “researcher degrees of freedom.”
The basic idea is straightforward once it’s pointed out. A single study almost never has just one obvious way to be analyzed. A researcher might choose among several plausible measures of the outcome they care about, several ways to define which participants count as outliers, several points at which to stop collecting more data, and several statistical tests that could reasonably be applied. Each of these choices is defensible on its own. But if a researcher, consciously or not, tries a handful of reasonable options and reports only the combination that produced a statistically significant result, the reported finding can look far more solid than it actually is. This practice came to be known informally as “p-hacking,” and demonstrations showed that it was possible to produce statistically significant support for obviously false claims simply by exploiting this kind of flexibility.
Related to this was the problem of publication bias, sometimes called the “file drawer” effect. Journals have historically shown a strong preference for publishing positive, novel, surprising results over null results or straightforward confirmations of existing knowledge. A study that finds no effect, or a study that simply confirms what everyone already expected, is less likely to be submitted, and if submitted, less likely to be accepted. Over time, this quietly filters the published literature so that it overrepresents chance findings and underrepresents the more boring, and often more accurate, null results that never saw daylight.
None of this required anyone to be dishonest. It required only that ordinary human judgment, operating under uncertainty and under pressure to produce interesting results, would tend to nudge choices in a particular direction without anyone fully noticing they were doing it.
The Psychology of Publish or Perish
It’s tempting to treat the reproducibility problem as a purely technical failure — bad statistics, fixable with better statistics. But the researchers who studied the crisis most closely kept returning to something more human: the incentive structure surrounding academic careers.
Hiring, promotion, tenure, and grant funding in most research institutions have long depended heavily on publication record, particularly publications in prestigious journals. Those journals, in turn, have historically favored novel, positive, and narratively clean findings over careful null results or unglamorous replications. A young researcher facing that landscape doesn’t need to be unethical to feel pulled toward analytic choices that produce a publishable result rather than an accurate one. The incentive isn’t “cheat.” The incentive is “keep looking until something works,” which feels, from the inside, like diligence rather than distortion.
This pattern is well documented in behavioral research on decision-making under incentive pressure: people are remarkably good at convincing themselves that a self-serving choice was the objectively correct one, especially when it arrives through a long chain of small, reasonable-seeming decisions rather than one obvious act of dishonesty. Several commentators argued the crisis was less about a few bad actors than about a system that had, gradually and without deliberate design, rewarded the appearance of discovery over its reliability — a pressure that fell hardest on early-career researchers, who had the least job security and the most to lose from a thin publication record. Describing an incentive structure that predictably produces certain behaviors is different from claiming individual scientists acted in bad faith, and the distinction is worth naming carefully rather than skipping over.
Building a Different Set of Incentives
By the middle of the 2010s, the response to all of this had moved from diagnosis to reform, and the reforms tended to target the incentive structure directly rather than simply exhorting individual researchers to be more careful.
One of the central ideas was preregistration: publicly stating, before collecting any data, exactly what hypothesis a study will test, how the sample will be determined, and how the data will be analyzed. Preregistration doesn’t prevent a study from failing to find an effect, but it closes off the kind of after-the-fact flexibility that made p-hacking possible, because the analysis plan is locked in before the results are known. A related innovation, the Registered Report, took this further by having journals review and provisionally accept a study based on its design and reasoning alone, before any data existed, with publication guaranteed regardless of whether the results turned out to be positive or null. This directly attacked publication bias, since a journal committed to publishing the outcome couldn’t later reject the paper simply because the finding was less exciting than hoped.
Open data practices grew alongside these changes, encouraging or requiring researchers to make their raw data and analysis code available so that other scientists could check the work directly rather than relying on trust in the published summary. For many researchers, that meant sharing analysis code through version-controlled public repositories, treating computational reproducibility — whether someone else’s software produces the same numbers from the same data — as inseparable from statistical reproducibility. Organizations like the Center for Open Science, founded in 2013 and closely associated with the Reproducibility Project: Psychology, built infrastructure — most notably the Open Science Framework — to make preregistration, data sharing, and collaborative replication projects easier to carry out in practice. A related set of standards, the Transparency and Openness Promotion guidelines, gave journals a shared framework for what they should require of authors around data availability and methodological disclosure.
None of these reforms were universally adopted, and they came with legitimate debate about cost, feasibility across different kinds of research, and whether they addressed the full range of disciplines affected. Preregistration works differently for a controlled laboratory experiment than for, say, long-term epidemiological research built on existing datasets. But by the second half of the 2010s, preregistration and open data had moved from a niche practice associated with a handful of methodological reformers into something increasingly expected by major journals and funding bodies across psychology, and to a lesser but growing extent, in biomedical research as well.
A More Complicated Picture Than “Science Is Broken”
It’s worth resisting a tempting but overly simple conclusion: that the reproducibility crisis proved science itself was untrustworthy. The more accurate reading, and the one most methodologists who studied the crisis actually reached, is closer to the opposite. Reproducibility problems exist in essentially every empirical field to some degree, but the fact that psychology and biomedical science went looking for these problems, measured them carefully, and built institutional responses is itself an example of science’s self-correcting machinery working as intended, if somewhat belatedly.
There was also legitimate pushback against some of the more sweeping claims made during the crisis’s most intense years. Some scholars argued that a single failed replication doesn’t cleanly disprove an original finding, since replications themselves can differ from the original study in ways that matter, or can simply be underpowered in a different direction. Others pointed out that framing the field’s discovery rate as evidence that “most findings are false” depends heavily on assumptions that are difficult to verify empirically. The debate over exactly how bad the problem was, and exactly which fixes mattered most, continued well past the initial wave of alarm, and reasonable methodologists disagreed with each other on the details even while agreeing that something needed to change.
What’s less disputed is the shift in norms that followed. Statistical training in graduate programs began incorporating discussions of researcher degrees of freedom and preregistration as standard material rather than niche methodology. Journals adjusted submission requirements. Funding agencies began asking more pointed questions about data management and analysis plans. Scientific culture rarely changes quickly, but by the mid-2010s, the vocabulary and expectations around rigor already looked noticeably different from a decade earlier, and they kept shifting in the years that followed.
Where the Story Leads
Looking back from 2026, the reproducibility challenge reads less like a scandal with a clean ending and more like the opening chapter of an ongoing, unglamorous project: building scientific infrastructure that makes honest mistakes easier to catch and dishonest or unconscious shortcuts harder to take. Large-scale replication efforts have continued across additional fields, including economics and other social sciences, extending the same basic method — repeat the study carefully, see what holds up — to test how widespread the underlying problem really was. Open data and preregistration, once optional signals of unusual rigor, have become closer to baseline expectations at many journals, even as debates continue about how to apply these standards consistently across very different kinds of research.
The deeper lesson of the period may have less to do with statistics than with institutional design. The reproducibility crisis showed how a system built by careful, well-intentioned people can still produce unreliable output when its incentives point the wrong way. Real change meant altering what got rewarded — publishing null results, sharing data openly, locking in an analysis plan before seeing the numbers — so that careful science and a successful career stopped pulling in opposite directions, at least some of the time. That recalibration is still underway, and remains one of the more consequential, if quiet, transformations in how modern science checks its own work.
Scientific Reference
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531–533.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Levelt Committee, Noort Committee, & Drenth Committee. (2012). Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Tilburg University, University of Amsterdam, and University of Groningen.
