Why the replication crisis led to one of the biggest methodological reforms in modern psychological science
Originally published in February of 2018 and comprehensively updated/edited in July 2026.
How psychology’s 2018 reporting reforms, from preregistration to open data, helped the field confront its replication crisis and rebuild trust in itself.
By the middle of the 2010s, psychology was having a very public identity crisis. For decades, the field had produced findings that shaped how millions of people understood themselves: why we lie, why we conform, why willpower seems to run out by the end of a long day. These ideas made their way into TED talks, bestselling books, corporate training seminars, and college syllabi. Then, one by one, many of them began to fall apart.
Researchers attempting to repeat classic experiments were getting different results, or no results at all. The phenomenon even had a name that stuck: the “replication crisis.” It wasn’t limited to psychology, but psychology became its most visible face, partly because so many of its findings had escaped the lab and settled into everyday life. When a psychological claim about human nature turns out to be shaky, it doesn’t just embarrass a handful of scientists. It unsettles something people thought they understood about themselves.
Researchers today often draw a more precise distinction within this broader problem. Reproducibility means getting the same results by reanalyzing the original data with the original methods, while replicability means getting similar results in an entirely new study with new participants. Both concerns fed into the same public conversation, but they point to slightly different failures, one about whether the original analysis was sound, the other about whether the underlying effect actually holds up beyond a single sample.
Those concerns eventually culminated in one of psychology’s most important methodological reforms: the APA’s revised Journal Article Reporting Standards, or JARS, released in January 2018. Before getting to what that reform actually involved, it’s worth understanding the conditions that made it necessary in the first place.
On the surface, a revised set of reporting guidelines sounds like a bureaucratic footnote: new rules about how researchers should write up their studies. In practice, it was one of the field’s most serious attempts to rebuild trust from the inside out. This article explores why that moment mattered, the problems it was designed to address, and how its reforms reshaped psychological science over the years that followed.
The Trouble That Led Here
To understand why reporting standards became such a big deal, it helps to understand what had gone wrong in the years before.
A Decade of Doubt
Psychology’s reputation problems didn’t appear overnight. Through the early 2010s, a series of events chipped away at public and professional confidence. A well-known Dutch social psychologist, Diederik Stapel, was found to have fabricated data in dozens of published studies. Around the same time, a large collaborative project called the Reproducibility Project attempted to redo about one hundred psychology experiments that had been published in top journals. Fewer than half produced the same results the second time around.
These weren’t obscure, forgettable studies. Some touched on ideas that had become part of popular culture, like the notion that adopting a “power pose” for two minutes could measurably shift a person’s hormones and confidence, or that reminding people of money makes them behave more selfishly. The idea of “ego depletion,” the theory that willpower functions like a limited fuel tank that gets used up over the course of a day, had inspired an entire cottage industry of advice books and productivity tips, until a large multi-lab replication effort found the effect was far weaker and less consistent than originally reported, if it was present at all.
When effects like these failed to reappear in careful follow-up studies, it raised an uncomfortable question, not just for scientists but for anyone who had absorbed these ideas as settled facts: how many other findings, quietly accepted for years and repeated in classrooms and self-help books, might be built on similarly shaky ground? The unease wasn’t really about any single study. It was about realizing that the filtering process separating solid findings from lucky flukes had never been as reliable as everyone assumed.
Why Findings Failed to Hold Up
The replication crisis wasn’t primarily a story about fraud, even though fraud cases like Stapel’s captured headlines. Most of the problem came from something less dramatic and more structural: the everyday habits of how research got done and published.
Journals overwhelmingly preferred to publish positive, exciting results over null or ambiguous ones. This created quiet pressure on researchers to find something significant, and gave rise to practices researchers later described bluntly as “p-hacking,” running many small variations of an analysis until one crossed the threshold of statistical significance, then reporting that one as if it had been the plan all along. Sample sizes in psychology experiments were also often small, sometimes just a few dozen participants, which made it easier for chance results to look like real effects. Another lesson to emerge from this period was a growing preference for reporting effect sizes and confidence intervals, measures of how large and how precisely estimated a finding actually is, rather than treating a result as simply significant or not based on whether it crossed the conventional p < .05 cutoff. None of this necessarily involved bad intentions. It reflected a scientific culture that rewarded novelty and confirmation over caution and replication.
What the New Standards Actually Changed
The APA’s 2018 revision to its reporting standards was the professional response to these structural problems. It didn’t rewrite psychological theory. Instead, it rewrote the expectations for how psychological findings should be documented, so that other scientists, and eventually the public, could tell the difference between a well-supported claim and a fragile one.
Making the Research Process Visible
One of the central goals of the new standards was transparency: giving readers enough detail about how a study was actually conducted that they could evaluate it fairly, and ideally repeat it. This meant clearer expectations around describing exactly how participants were recruited, what measures were used and why, and how many analyses were run before landing on the ones that got published.
This last point mattered enormously. If a research team quietly tested ten different ways of measuring an outcome and only reported the one that “worked,” readers had no way of knowing that from the published paper alone. The new standards pushed toward disclosing that kind of decision-making, treating the research process itself as something worth documenting honestly, not just the final tidy result.
Preregistration: Committing Before You Peek
Perhaps the most influential idea folded into the 2018 standards was preregistration. In simple terms, preregistration means a researcher publicly states, before collecting or analyzing data, exactly what they expect to find and how they’ll test it. That plan gets time-stamped and stored somewhere outside the researcher’s control, often through an independent registry.
The logic behind this is straightforward once you see it. If you decide in advance what counts as a meaningful result, you can’t quietly shift your analysis afterward to manufacture one. It’s the difference between predicting where a dart will land before you throw it and drawing a target around wherever it happens to stick. Preregistration doesn’t make research boring or rigid; it simply closes the door on a very human temptation to see patterns after the fact and mistake them for a discovery someone planned to test.
Much of this infrastructure didn’t have to be built from scratch. A nonprofit called the Center for Open Science had already launched a free platform, the Open Science Framework, where researchers could register their study plans, store their materials, and later post their raw data for anyone to examine. By the time the 2018 standards arrived, the Open Science Framework had become something close to the default home for preregistration in psychology, giving the new guidelines a practical tool to point to rather than leaving methodological openness as an abstract ideal. Sharing raw data openly, alongside a published paper, became just as central to this shift as preregistering a hypothesis. Where earlier journal articles typically reported only summary statistics and final conclusions, open data practices let other researchers examine the underlying numbers directly, checking not just what a study concluded but whether the conclusion actually followed from the data itself.
Widening the Toolkit: Statistics Built for Real-World Complexity
The 2018 standards also expanded formal guidance for research designs and statistical approaches that had grown more common but lacked consistent reporting norms. This included clinical trials, long-term studies that follow the same people over months or years, and purely observational research where scientists study behavior as it naturally occurs rather than manipulating anything in a lab.
It also opened the door wider to two more sophisticated statistical approaches: Bayesian statistics and structural equation modeling. Without getting lost in technical detail, Bayesian methods offer a different way of thinking about evidence. Instead of asking “is this result statistically significant, yes or no,” Bayesian analysis asks “given what we already knew, how much should this new evidence update our confidence?” It treats knowledge as something that shifts gradually rather than flipping from unproven to proven in a single study. Structural equation modeling, meanwhile, is a technique for testing complicated webs of cause and effect, useful for questions where several psychological factors influence each other simultaneously rather than lining up in a single straight cause-to-effect chain. Formally recognizing these approaches gave researchers more honest tools for representing how messy human behavior actually is, instead of forcing every question into a simplified yes-or-no test.
From Rule Change to Culture Shift
New guidelines on paper are one thing. Whether they actually change how science gets done is another question entirely, and this is where the 2018 reforms proved to have real staying power.
Replication Becomes Respectable
Before the crisis, repeating someone else’s study to check if it held up carried little professional reward. Journals wanted new discoveries, not confirmations, and researchers built careers on novelty. The reforms coincided with a broader cultural shift within the field: replication studies began appearing in respected journals, and doing careful, unglamorous confirmatory work stopped being seen as a lesser form of science.
One of the biggest innovations to emerge from this shift was the registered report. Instead of judging research based on whether the results turned out exciting, a growing number of journals began evaluating studies before data collection even started. If the proposed methods were sound and the question worth asking, publication was provisionally accepted regardless of whether the eventual findings supported the hypothesis or not. That single change struck directly at the root of publication bias. A researcher no longer had any incentive to chase a flashy result or bury a disappointing one, because the decision to publish had already been made before anyone knew how the data would turn out.
This shift reflected something psychologically interesting in its own right. Fields, like individuals, develop habits and incentives that shape behavior over time. When the incentive structure rewarded flashy, surprising findings, researchers, entirely rationally, produced more flashy, surprising findings, some of which turned out not to be real. When the incentives shifted toward reporting clarity and verification, behavior shifted with them. It was a fairly direct demonstration of one of psychology’s own core ideas: that context and consequence shape conduct, whether the people involved are laboratory participants or the scientists studying them.
A Field Learning to Audit Itself
There’s a certain irony, and a kind of maturity, in a discipline devoted to studying human judgment and bias turning that same lens on its own professional practices. Cognitive biases like confirmation bias, the tendency to notice and favor evidence that fits what we already believe, aren’t unique to laypeople. Trained scientists are just as capable of unconsciously steering their analyses toward the outcome they hoped for. The 2018 reforms amounted to psychology acknowledging that its own researchers were subject to the same mental shortcuts they’d spent decades documenting in everyone else.
That kind of institutional self-scrutiny doesn’t happen automatically. It required public embarrassment, sustained internal debate, and pressure from a rising generation of researchers, many trained after the replication crisis broke, who treated open practices as the baseline expectation rather than an optional extra.
Why This Mattered Beyond the Laboratory
It would be easy to treat this as a story confined to academic journals, but the stakes reached well past university psychology departments.
Public Trust and the Cost of Overreach
Psychological findings shape real decisions. They influence how teachers structure classrooms, how therapists choose treatments, how companies design workplace policies, and how parents think about raising children. When a widely cited finding later turns out to be unreliable, the damage isn’t contained to a single retracted paper. It quietly erodes public willingness to trust behavioral science more broadly, making people more skeptical even of findings that are genuinely well supported.
The push for preregistration, open data, larger samples, and clearer reporting was, at its heart, an attempt to protect that trust before it eroded further. Science earns its authority not by being infallible, but by having built-in mechanisms for catching and correcting its own mistakes. The 2018 standards were an attempt to make those mechanisms sturdier.
Consider how far a single study can travel once it leaves the lab. A finding about willpower or body language gets picked up by a journalist, distilled into a headline, repeated in a corporate seminar, and eventually taught as accepted wisdom in a college lecture, all without most of the people passing it along ever reading the original methodology. That’s not a flaw unique to psychology; it’s simply how ideas move through a culture. But it does mean that weaknesses in the original research get amplified at every step, while the eventual failed replication, published years later in a much quieter journal, rarely travels nearly as far. Tightening the standards at the source, before a shaky finding ever gets the chance to go viral, was a far more effective fix than trying to correct the record after the fact.
A Model for Other Fields
Psychology wasn’t alone in facing this reckoning, but it moved relatively early and relatively visibly. Other disciplines grappling with their own reproducibility concerns, including parts of biomedical research and some corners of economics, looked to psychology’s reforms as a partial template. The specific tools varied by field, but the underlying principle traveled well: research becomes more trustworthy when the process behind it is visible, and when confirming a finding is valued nearly as highly as discovering one in the first place.
Looking Back From 2026
Nearly a decade on, it’s possible to see the 2018 reporting standards not as an isolated event but as one clear marker along a longer arc of reform. Preregistration, once a niche practice mostly discussed among methodology specialists, is now a routine expectation at many major journals. Registered reports, where a study’s methods are peer-reviewed and provisionally accepted before the results are known, have expanded steadily across psychology and into adjacent fields. Open data repositories, where researchers share their raw data and analysis code alongside a published paper, have become far more common than they were in 2018, allowing outside researchers to check a finding’s foundations directly rather than taking the summary on faith.
None of this means psychology’s credibility problems are fully solved. Incentive structures in academic publishing still favor certain kinds of results, career pressures on researchers haven’t disappeared, and new methodological debates, particularly around how findings generalize across different populations and cultures, have taken center stage in the years since. But the conversation itself changed. Scientific accountability, replication, and honest reporting of uncertainty are no longer treated as optional virtues for the unusually scrupulous scientist. They’re closer to baseline professional expectations, reinforced by journal policy, funding requirements, and a research culture that came of age during the reckoning itself.
Conclusion: A Discipline Rebuilding Its Own Foundation
The APA’s 2018 revision of its reporting standards was never going to be a dramatic headline on its own. It was a technical document about how studies should be written up, not a discovery about the human mind. But its significance lay precisely in that quiet, structural nature. It represented psychology choosing to address its credibility crisis not by dismissing the criticism or hoping it would fade, but by rebuilding the scaffolding underneath its own claims.
The deeper lesson extends past psychology itself. Confidence in any field of knowledge doesn’t rest on the promise that its practitioners are never wrong. It rests on whether the field has honest, working mechanisms for finding out when it is wrong and correcting course. In 2018, psychology took a meaningful step toward strengthening those mechanisms. Looking back from 2026, with preregistration and open methods now woven into the field’s everyday practice, that step looks less like a single turning point and more like the start of a discipline that finally learned to check its own work.
Ironically, the replication crisis may ultimately be remembered not as the moment psychology lost credibility, but as the moment it demonstrated the self-correcting process that gives science its credibility in the first place.
Scientific References
American Psychological Association. (2018). Journal Article Reporting Standards (JARS). APA Publications and Communications Board Working Group.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251).
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606.
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29.
