Originally published in September 2013 and comprehensively updated/edited in July 2026.
How fraud scandals, failed replications, and statistical problems triggered the open science movement and transformed how researchers share data, design studies, and validate discoveries.
In the fall of 2011, a Dutch social psychologist named Diederik Stapel confessed that he had spent years fabricating data. This was not a case of selectively emphasizing favorable results or making a questionable interpretation. In some cases, he had invented entire experiments, complete with fictional participants and fictional results, then published the invented findings in respected journals. The investigation ultimately led to the retraction of dozens of his papers. It was the kind of scandal that could have been dismissed as an isolated case of one dishonest man. Instead, it became a symptom of something much larger, and much harder to fix.
That same year, a Cornell psychologist named Daryl Bem published a paper in a top-tier journal claiming to have found statistical evidence that people could sense future events before they happened. The methods looked ordinary. The statistics were, by the conventions of the time, entirely acceptable. And yet almost no one in the field believed the conclusion. If precognition wasn’t real, but the paper had passed peer review anyway, then something was wrong not with Bem specifically, but with the rules everyone was playing by. The point was not that peer review had failed because it allowed an unusual idea to be published; science needs room for unconventional hypotheses. The deeper problem was that conventional statistical standards were capable of producing confidence in findings that later appeared unreliable.
These two episodes, arriving within months of each other, crystallized a suspicion that had been building quietly for years: a lot of published science, especially in fields like psychology, medicine, and the social sciences, might not hold up if anyone bothered to check it. Researchers began to check. What they found, over the following several years, became known as the replication crisis, and the response to it became known as the open science movement. This is the story of how science tried to catch itself making mistakes, and what it built in order to do so.
A Discipline Built on Trust, Strained by Incentives
To understand why the crisis happened, it helps to understand how science had been working for decades. A researcher had an idea, ran a study, analyzed the results, and wrote up whatever turned out to be interesting. Journals then decided what counted as interesting enough to publish. This system was never a secret conspiracy. It was simply how things had always been done, and it rested on an assumption that most people found reasonable: that scientists, trained to pursue truth, would report what they found honestly.
The trouble was that the system rewarded a particular kind of finding. A result that was clean, novel, and statistically significant was far more publishable than a messy result or a study that found nothing at all. Career advancement, tenure, and grant funding all depended heavily on publication record, and journals overwhelmingly preferred positive results over null ones. Over time, this created what researchers began calling publication bias, a systematic tilt in the scientific literature toward findings that confirmed something interesting, and away from findings that didn’t.
This tilt did not require anyone to lie. It could operate through much subtler mechanisms. A researcher running an experiment might test several related hypotheses, several ways of measuring the outcome, and several subsets of the data, then report only the analysis that produced a significant result while treating the others as false starts. This practice eventually acquired a name: p-hacking, a reference to the p-value, the statistical measure researchers use to decide whether a result is likely to be real rather than due to chance. Done unconsciously, and it usually was unconscious, p-hacking could turn ordinary statistical noise into what looked like a genuine discovery.
A related problem involved the order in which hypotheses and results were presented. A researcher might run a broad, exploratory study, notice an interesting pattern buried in the data, and then write the paper as though that pattern had been the prediction all along. This came to be called HARKing, short for Hypothesizing After the Results are Known. It is not fraud in the legal sense. It is something closer to a storytelling habit, one that made science look far more predictive, and far more reliable, than it actually was.
None of this was new in 2011. A prominent epidemiologist named John Ioannidis had argued in a widely read 2005 paper that a great many published research findings were probably false, for reasons rooted in exactly these statistical and structural incentives. But Ioannidis’s warning had remained largely theoretical to most working scientists until Stapel and Bem gave the field two vivid, undeniable examples of what the theory predicted.
The Reproducibility Project and the Scale of the Problem
Suspicion is one thing. Evidence is another. If the scientific literature was full of unreliable findings, the only way to know for certain was to test it directly, by taking published studies and trying to repeat them under the same conditions to see whether they produced the same results.
This is precisely what a large, loosely organized group of researchers set out to do beginning around 2011, under a project that would eventually be called the Reproducibility Project: Psychology. Coordinated in large part by a University of Virginia psychologist named Brian Nosek, the project recruited roughly two hundred researchers from around the world to each replicate one study from three major psychology journals, following the original methods as closely as possible.
The results, published in the journal Science in 2015, were sobering. Fewer than half of the one hundred replicated studies produced a statistically significant result matching the original. Even among those that did replicate, the size of the effect was, on average, about half as large as originally reported. Psychology was not unique in this respect; subsequent large-scale replication efforts in fields ranging from cancer biology to economics turned up broadly similar patterns, though the exact numbers varied by discipline. The message across all of them was consistent. A surprising number of influential findings in the scientific literature had never been tested a second time, and when they finally were, many failed to reproduce at their original strength.
It’s worth pausing on what a failed replication does and does not mean. A study that fails to replicate is not automatically proof of fraud, and in the overwhelming majority of cases fraud plays no role at all. Failed replications can result from small and genuine differences between the original study and the repeat attempt, from studies that were too small to detect a real but modest effect in the first place, from unlucky statistical variation, or from an original result that was a false positive produced by exactly the p-hacking and HARKing described above. Distinguishing between these explanations, case by case, is difficult and often impossible after the fact. That difficulty was itself part of the argument for reform. If a field couldn’t tell, after publication, whether a given finding was solid, then the fix needed to happen before publication, at the level of how research was designed, conducted, and recorded in the first place.
Preregistration: Locking In the Plan Before Seeing the Data
The most direct response to HARKing and p-hacking was a practice borrowed, in part, from clinical drug trials, where regulators had long required researchers to state in advance exactly what they intended to test. This practice is called preregistration.
In a preregistered study, a researcher writes down the hypothesis, the planned sample size, the way the data will be measured, and the statistical analysis that will be used, and submits this document to a public registry before collecting any data at all. The document is time-stamped and, in most cases, made publicly viewable. If the study later gets written up for publication, readers can compare the final paper against the original plan and see immediately whether the researcher stuck to it or quietly changed course after seeing which analysis produced the cleanest result.
Preregistration doesn’t stop a researcher from exploring their data in creative, unplanned ways after the fact. It simply requires that exploratory findings be labeled as exploratory rather than dressed up as confirmed predictions. This distinction, between confirmatory research that tests a specific pre-stated hypothesis and exploratory research that generates new hypotheses for future testing, became one of the central organizing ideas of the open science movement. Both kinds of research are valuable. The problem, reformers argued, was never exploration itself, but exploration disguised as confirmation.
Some journals went a step further and adopted a format called Registered Reports, in which a study’s introduction and proposed methods are peer reviewed and provisionally accepted before the data are even collected. Because the decision to publish is made before anyone knows the outcome, this format removes publication bias almost by design; a well-designed study that finds nothing gets published exactly as readily as one that finds something dramatic.
The Infrastructure of Openness
Good intentions needed tools, and the years following 2011 saw a wave of new infrastructure built specifically to support these practices. The most significant piece of that infrastructure was the Open Science Framework, a free online platform launched in November 2012 by the newly formed Center for Open Science, a nonprofit that Nosek co-founded with fellow researcher Jeffrey Spies in Charlottesville, Virginia. The Open Science Framework gave researchers a single place to register their study plans, store their raw data, share their analysis code, and document the entire lifecycle of a project in a form that other scientists could inspect. Adoption started slowly and then accelerated for more than a decade; what began as a niche tool for early adopters had, by the mid-2020s, become a major piece of infrastructure for researchers practicing open science worldwide.
Data sharing became the second pillar of the movement, alongside preregistration. Historically, the data underlying a published study often existed only on the original researcher’s personal computer, available to other scientists only if they happened to ask nicely and the original researcher happened to still have it. Open science advocates pushed for a different default: that data, along with the code used to analyze it, be deposited somewhere public and permanent at the time of publication, so that any other researcher could, in principle, rerun the original analysis and check it independently. Some journals began requiring this as a condition of publication. Some funding agencies began requiring it as a condition of a grant.
By 2015, the Center for Open Science had formalized much of this thinking into a document called the Transparency and Openness Promotion guidelines, a set of standards that journals could voluntarily adopt, organized into tiers so that a publication could commit to as much or as little transparency as it was ready for. Hundreds of journals eventually signed on in some form, turning what had started as the informal preference of a subset of reform-minded researchers into something closer to an institutional expectation.
Why the Crisis Cut So Deep, Psychologically
There is a reason the replication crisis provoked such a strong emotional reaction within the scientific community, beyond the purely technical problem of unreliable statistics. Science, as an institution, draws much of its cultural authority from the claim that it is self-correcting, that any error will eventually be caught and fixed by the community’s own methods. The replication crisis suggested that this self-correction had, for decades, been happening far more slowly and far less completely than anyone assumed. That is a difficult thing for a professional community to sit with, because it touches identity as much as method.
Behavioral scientists who studied the crisis from the outside noted that many of the psychological patterns driving bad research practices were the same patterns their own field had spent decades documenting in other contexts: confirmation bias, the tendency to notice and remember evidence that supports what we already believe; motivated reasoning, the tendency to reach conclusions we want to reach and then construct a rationale for them after the fact; and a simple, understandable reluctance to abandon a hypothesis a researcher had invested years of their career pursuing. None of this required deliberate dishonesty. It required only the ordinary cognitive habits of ordinary people, operating inside an incentive structure that happened to reward those habits rather than correct for them.
This reframing mattered because it shifted the conversation away from blame and toward system design. If the problem was mostly structural, then the solution had to be structural too. Preregistration worked, when it worked, not because it made researchers more virtuous, but because it removed the opportunity to unconsciously drift toward a favorable analysis after the data were already in hand. It changed the environment rather than asking individuals to simply try harder to resist a bias that operates, by definition, below conscious awareness.
Resistance and Growing Pains
The movement did not spread without friction. Some senior researchers, including a few whose own work came under replication scrutiny, argued that the new emphasis on preregistration and replication threatened to punish creativity and exploratory science, the kind of open-ended curiosity that had produced many of the field’s most important discoveries in the first place. Others pointed out, reasonably, that replication attempts themselves could be poorly executed, and that a failed replication did not always mean the original finding was wrong.
There were also uncomfortable moments where reform efforts collided with individual careers. Public failed replications sometimes named the original researchers, and social media discussion of those replications could turn considerably less measured than the underlying scientific disagreement warranted. Some researchers who had built careers on findings that later failed to replicate described the experience as genuinely difficult, professionally and personally, regardless of whether any wrongdoing had occurred on their part. The movement’s advocates generally tried to draw a firm line between criticizing a finding and criticizing the person who produced it, though that line was not always respected in practice, particularly online.
Younger researchers, notably, often embraced the new norms more readily than established ones, in part because they were building their careers within the new system rather than being asked to retroactively justify decades of prior work made under different rules. Graduate programs began teaching preregistration and open data practices as standard methodology rather than as optional reform, which meant the changes were, to a significant degree, generational as well as institutional.
Beyond Psychology
Although psychology became the field most associated with the replication crisis, in part because it was the discipline that investigated the problem most aggressively and publicly, the underlying issues were never unique to it. Cancer biology, a field with obvious life-or-death stakes, produced its own large-scale replication effort, the Reproducibility Project: Cancer Biology, launched around the same period and coordinated through a partnership with the company Science Exchange. Economics, political science, and biomedicine each developed their own versions of preregistration and replication culture over the following decade, adapted to the particular methods and constraints of each field. Clinical trial registries, which had existed in a more limited form since the 1990s for regulatory reasons, expanded and tightened their requirements in ways that echoed the broader open science push.
By the early 2020s, funding agencies including the National Institutes of Health and the National Science Foundation had incorporated data-sharing requirements and encouraged or required greater transparency practices in many areas of research funding, which meant the movement’s ideas had moved well beyond the voluntary efforts of individual reform-minded scientists and into the formal machinery that determines what research gets funded at all.
What the Movement Changed, and What It Didn’t
It would be an overstatement to say the open science movement solved the problems it set out to address. Fabrication cases still surface periodically. Publication bias has weakened in fields that adopted registered reports and similar formats, but it has not disappeared, and plenty of journals never adopted these formats at all. Preregistration, done poorly, can become a box-checking exercise rather than a genuine constraint, especially when researchers write vague, flexible preregistrations that leave room for the same after-the-fact interpretive freedom the practice was designed to eliminate.
What did change, more durably, was the default expectation among a large and growing share of the scientific community. In 2011, sharing your raw data, or stating your hypothesis in advance and sticking to it, was an unusual choice that a researcher had to actively decide to make, usually for idiosyncratic reasons. By the mid-2020s, in many fields, the reverse was true: not sharing data or not preregistering a confirmatory study had itself become the choice that required explanation. That kind of shift in what counts as the ordinary, unremarkable way of doing things is slow, uneven, and hard to measure precisely, but it is also, historically, how scientific norms actually change. Not through a single decisive reform, but through a gradual redrawing of what looks normal to the next generation of researchers walking into a lab for the first time.
Looking Forward
The open science movement that took shape after 2011 was, at its core, a response to a specific and uncomfortable realization: that the incentives guiding individual scientists did not automatically align with the collective goal of building reliable knowledge, and that closing that gap would require deliberate changes to infrastructure, publishing norms, and researcher training rather than an appeal to individual integrity alone. Preregistration, open data, and large-scale replication projects were the movement’s most visible tools, but the underlying insight extended further, into how institutions structure the incentives faced by the people who work inside them.
The lasting legacy of the open science movement may not be any single tool it created. It may be the recognition that science itself is a human institution, and like every human institution, it must be deliberately designed to compensate for the weaknesses of the people inside it.
Scientific Reference
Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Nosek, B. A., et al. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425.
Errington, T. M., et al. (2021). Investigating the replicability of preclinical cancer biology. eLife, 10, e71601.
Center for Open Science. (n.d.). About the Center for Open Science and the Open Science Framework. Retrieved 2026, from cos.io.
