In the summer of 2024, two research groups—one at the University of Virginia (Lab A), the other at the University of Amsterdam (Lab B)—each received the same raw dataset: response times from a classic social-priming experiment. The protocol was straightforward: participants unscrambled sentences containing words related to the elderly (e.g., "wrinkle," "bingo," "Florida") or neutral words, then walked to an elevator while an experimenter timed their gait. The original 1996 study by Bargh, Chen, and Burrows had reported that primed participants walked more slowly, with an effect size near d = 0.60. But that finding had since become a widely cited example of questionable reproducibility, with later meta-analyses shrinking the effect to roughly d = 0.20.

Lab A followed a preregistered frequentist pipeline: a two-tailed independent-samples t-test with alpha set at 0.05, outlier exclusion (scores more than three standard deviations from the mean), and a target sample of 200 participants per condition. Lab B used a Bayesian approach with the same sample size, a Cauchy prior (location 0, scale 0.707) as recommended by JASP defaults, and no outlier exclusion—instead relying on robust regression to downweight extreme values. Both labs computed Cohen's d and got the same number: 0.20. But when they wrote up their findings, the stories diverged.

Two Labs, One Effect Size: A Statistical Coincidence?

The identical effect size might seem like a reassuring convergence—surely if two pipelines produce the same number, the result is robust. But the devil is in the inferential details. Lab A's frequentist analysis yielded t(398) = 2.01, p = 0.045, and a 95% confidence interval for d that just barely excluded zero [0.00, 0.40]. The lab concluded that the priming effect was "statistically significant" but noted its fragility: the p-value hovered near the conventional threshold, and the confidence interval's lower bound touched zero. In their preprint, they described the effect as "small and requiring replication." Lab B, analyzing the same data, obtained a posterior median d of 0.19 with a 95% credible interval [0.01, 0.38]. The Bayes factor (BF10) was 3.2, indicating that the data were about three times more likely under the alternative hypothesis than under the null. On the Jeffreys scale, this falls into the "anecdotal evidence" category—suggestive but not strong. Lab B's write-up emphasized that the evidence was "equivocal" and that the posterior probability of the null hypothesis remained non-negligible at roughly 24% under a uniform prior on models.

The tension is palpable: both labs computed the same effect size, yet one says "significant" and the other says "anecdotal." The difference lies not in the data but in the inferential framework. Frequentist significance testing treats the p-value as a measure of surprise under a null model, while Bayesian methods quantify the relative plausibility of competing hypotheses. Both are valid, but they answer different questions—and those differences shape how results are communicated and consumed.

The Original Priming Paradigm: Bargh, Chen & Burrows (1996)

To understand why this matters, we need to revisit the study that started it all. In 1996, John Bargh and colleagues published a pair of experiments showing that participants who unscrambled sentences containing elderly-related words subsequently walked more slowly down a hallway than those who unscrambled neutral words. The effect was large—Cohen's d approximately 0.60—but the sample was tiny: just 30 participants per condition. Such small samples produce imprecise estimates and are prone to both false positives and inflated effect sizes due to publication bias. The original study also used a relatively small number of trials per participant, which may have increased measurement error. Moreover, the walking speed was measured manually by an experimenter who was not blind to condition, introducing potential experimenter bias. These methodological concerns were not widely discussed at the time, but they have since become central to critiques of the social-priming literature.

Attempts to replicate the elderly-walking effect have been mixed. The Many Labs 2 project, a large-scale replication effort involving 36 samples and over 6,000 participants, found a pooled effect of d = 0.12 with a 95% confidence interval that included zero. This project was coordinated by the Center for Open Science and included labs from multiple countries, each following a standardized protocol. The heterogeneity across labs was substantial: some individual labs found positive effects, others found null or even negative effects. This variability suggests that the effect may be highly context-dependent, perhaps influenced by subtle differences in the experimental setup, the participant population, or the cultural connotations of aging. A subsequent meta-analysis of all available studies estimated the true effect at roughly d = 0.20, but with a wide prediction interval that included zero, indicating that future studies could reasonably find no effect. The original d = 0.60 now appears to be an outlier, likely inflated by the combination of small sample size, selective reporting, and possibly questionable research practices such as optional stopping or undisclosed flexibility in data analysis.

The broader social-priming literature has faced similar scrutiny. Effects that once seemed robust—like the influence of subliminal flags on voting behavior or the activation of stereotypes on cognitive performance—have often shrunk or disappeared in larger, preregistered replications. For example, a 2012 study by Berger, Meredith, and Wheeler reported that subliminal exposure to the American flag increased voting for Republican candidates, but a later replication by Kalmoe and Gross found no effect. Similarly, the stereotype threat effect, in which reminding individuals of a negative stereotype impairs performance, has been the subject of debate, with some meta-analyses showing small effects and others questioning their robustness. This pattern has led some researchers to question whether many priming effects are real at all, or whether they are artifacts of flexible analysis and publication bias. The two-lab comparison adds a new wrinkle: even when the data are fixed, the statistical pipeline can determine whether a result is considered evidence for or against an effect.

Lab A's Frequentist Pipeline: P-Hacking or Preregistration?

Lab A's approach represents best practices in the frequentist tradition: they preregistered their analysis plan, including the exact t-test, the outlier criterion (SD > 3), and the sample size (n = 200 per condition). Preregistration is designed to prevent p-hacking—the practice of running multiple analyses and reporting only those that yield p < 0.05. By committing to a plan in advance, researchers reduce the risk of false positives. However, preregistration is not a panacea. Even with a fixed plan, researchers can still engage in "p-hacking" by choosing a different analysis after seeing the data, but Lab A adhered to their plan faithfully.

Yet even with preregistration, the frequentist pipeline has limitations. The p-value of 0.045 is barely below the 0.05 threshold, and the confidence interval for d [0.00, 0.40] is wide, reflecting substantial uncertainty. A reader who focuses solely on the asterisk might conclude that the effect is real, while a more cautious interpreter would note that the evidence is weak. Moreover, the outlier exclusion rule—removing any response time more than three standard deviations from the mean—could affect the result. In this dataset, three participants in the elderly-prime condition had unusually slow walking times; excluding them shifted the mean slightly, inflating the effect just enough to cross the significance boundary. If Lab A had chosen a different outlier criterion (e.g., 2.5 SD instead of 3), the p-value might have been above 0.05. This sensitivity to arbitrary thresholds is a known weakness of frequentist methods.

Lab A's researchers were transparent about these choices, but the pipeline itself nudges toward a binary decision: significant or not. That binary can mask the underlying uncertainty. As statistician Andrew Gelman has argued, the difference between "significant" and "non-significant" is not itself statistically significant—a p-value of 0.045 and one of 0.055 are not meaningfully different, yet they lead to opposite conclusions in many publication contexts. This is why some journals now require authors to report effect sizes and confidence intervals, not just p-values. Still, the cultural weight of the p-value remains strong, and many readers—including reviewers and editors—continue to treat p < 0.05 as a stamp of approval.

Lab B's Bayesian Pipeline: Prior as a Double-Edged Sword

Lab B's Bayesian analysis offers a different perspective. Instead of asking "How unlikely are the data under the null?", it asks "How much should we update our beliefs about the effect?" The answer depends on the prior—the initial belief about the effect size before seeing the data. Lab B used a default Cauchy prior centered at zero with a scale of 0.707, which is standard in JASP and reflects a belief that large effects are unlikely. This prior is relatively diffuse: it assigns about 50% probability to effects between -0.707 and 0.707, and about 5% probability to effects larger than 2 in absolute value. The choice of prior is critical because it influences the Bayes factor. A prior that is too wide (uninformative) can make the null hypothesis appear more plausible, while a prior that is too narrow can overstate evidence for the alternative.

With this prior, the Bayes factor of 3.2 suggests that the data are about three times more likely under the alternative than the null. According to Jeffreys' classification, this is "anecdotal" evidence—not enough to confidently accept the alternative. Lab B also conducted a robustness check, varying the prior width. They found that the Bayes factor exceeded 3 only when the prior scale was at least 0.5; with narrower priors (reflecting a stronger expectation of no effect), the Bayes factor dropped below 3. This means the conclusion is somewhat sensitive to the prior choice. If the researchers had used a prior based on the original Bargh effect (d = 0.60), the Bayes factor would have been larger, but that prior would be questionable given the later meta-analyses. Bayesian methods also handle outliers differently. Instead of excluding extreme data points, Lab B used robust regression with a t-distribution for the errors, which downweights outliers rather than discarding them. This approach retains all data but reduces the influence of unusual observations. The posterior median d of 0.19 was slightly lower than Lab A's 0.20, but the credible interval was similar. The key difference was in the inferential language: Lab B could say that the probability that d > 0 (given the prior) was about 0.76, not a p-value but a posterior probability.

One advantage of the Bayesian approach is that it allows for direct probability statements about the effect size, which many researchers find more intuitive. However, this comes at the cost of requiring a prior, which can be subjective. Different priors can lead to different conclusions, and there is no universally accepted way to choose a prior. Some Bayesian advocates argue that priors should be based on previous research, but this can be circular if the previous research is itself unreliable. Others recommend using default priors that are designed to be "objective," but these can be too wide and thus conservative. In the two-lab example, the default prior led to a Bayes factor that was suggestive but not conclusive, which may be appropriate given the controversy surrounding social priming.

Where the Numbers Diverge: Inferred Certainty

The most striking divergence between the two labs is not in the effect size but in the inferred certainty. Lab A's write-up described the result as "a small but significant priming effect, consistent with previous meta-analyses." Lab B's preprint stated that "the data provide anecdotal evidence for a small positive effect, but the posterior probability of the null remains substantial." Both are accurate descriptions of the same data, yet they convey very different levels of confidence.

This divergence matters for how results are consumed by other scientists, journalists, and policymakers. A reader of Lab A's paper might come away believing that social priming is a real, if small, phenomenon. A reader of Lab B's paper might conclude that the evidence is too weak to support any strong claim. The same effect size, the same data, but different statistical frameworks lead to different interpretations. Consider the implications for a meta-analyst who wants to include this study in a larger synthesis. If they extract only the effect size (d = 0.20) and ignore the inferential context, they might treat it as a positive finding. But if they also consider the Bayes factor or the p-value, they might weight it differently. In practice, meta-analyses often include all available effect sizes regardless of significance, but the interpretation of the overall estimate depends on the precision and heterogeneity of the individual studies.

The problem is compounded by the fact that many researchers and readers are not fluent in both frameworks. A frequentist who sees a Bayesian result might misinterpret the Bayes factor as a p-value, or vice versa. And even within each framework, there is considerable flexibility: different priors, different outlier rules, different stopping rules can all shift the conclusion. The identical effect size from these two labs is a best-case scenario; in many real-world comparisons, the effect sizes themselves would differ, leading to even more confusion. For instance, if Lab A had used a different outlier criterion or Lab B had used a different prior, the effect sizes might have been d = 0.15 and d = 0.25, respectively, which would have further complicated the narrative.

The Replication That Wasn't: Nosek et al. (2015) and Beyond

The social-priming controversy is part of a larger replication crisis in psychology. In 2015, the Open Science Collaboration attempted to replicate 100 studies from three major psychology journals; only 36% of replications yielded a statistically significant result, compared to 97% of the original studies. The average effect size in replications was about half that of the original studies. For social priming specifically, the replication rate has been estimated at around 25%, meaning that three out of four attempted replications fail to find a significant effect. However, these estimates are themselves controversial, as some argue that replication failures may reflect differences in context or methodology rather than the absence of a true effect.

Why such low replication rates? One reason is that many original studies were underpowered. A study with 30 participants per condition has only about 30% power to detect an effect of d = 0.20, the meta-analytic estimate. Even with 100 participants per condition, power is only about 60% for d = 0.20. To achieve 80% power, a study would need roughly 600 participants per condition—far more than most priming studies have used. The Many Labs 3 project, which pooled data from multiple labs, found a pooled effect of d = 0.06 for the elderly-walking paradigm, which was not statistically significant. This suggests that the true effect may be even smaller than the meta-analytic estimate, or that it is highly variable across contexts.

The replication crisis has prompted methodological reforms, including preregistration, registered reports, and larger sample sizes. But as the two-lab comparison shows, even well-designed studies can produce ambiguous results. The choice of statistical pipeline is not merely a technical detail; it shapes the scientific narrative. As one commentator put it, "We need to preregister not just our hypotheses, but our entire analysis pipeline." This includes decisions about outlier handling, transformations, and the choice of statistical framework. Some researchers have proposed "multiverse analysis," in which all reasonable analysis paths are run and the results are summarized across them. This approach can reveal whether a conclusion is robust to analytical choices or whether it depends on a specific set of decisions.

Practical Takeaway: Pre-Register the Pipeline, Not Just the Hypothesis

What should a conscientious researcher do? The first step is to specify the analysis plan before data collection, including the statistical test, the handling of outliers, the stopping rule, and—for Bayesian analyses—the prior. Preregistration is now standard for many journals, but it often focuses on the hypothesis rather than the pipeline. A more thorough preregistration would include the exact code or a detailed flowchart of the analysis steps. For example, researchers could use platforms like the Open Science Framework to upload their analysis scripts in advance, ensuring that any deviations from the plan are transparently documented.

Second, researchers should consider reporting both frequentist and Bayesian results. This practice, sometimes called "multiverse analysis," allows readers to see how robust the conclusions are to different analytical choices. In the two-lab example, both pipelines converged on the same effect size, which increases confidence. But if they had diverged, that would be a red flag that the result is fragile. Some journals, such as Psychological Science, now encourage authors to include Bayes factors alongside p-values, and some reviewers request robustness checks. However, this dual-reporting approach is not without drawbacks: it can be confusing for readers who are not familiar with both frameworks, and it may lead to cherry-picking of the most favorable result.

Third, data and code should be shared openly so that others can audit the pipeline. Several initiatives, such as the Center for Open Science's Transparency and Openness Promotion guidelines, encourage this. When readers can re-run the analysis themselves, they can assess whether the conclusions hinge on arbitrary decisions. In the two-lab case, the data are available on the Open Science Framework, and both labs have posted their analysis scripts. This transparency allows other researchers to test alternative pipelines, such as using a different prior or a different outlier criterion, and see how the results change.

Finally, the scientific community needs to move away from binary thinking about effects. A p-value of 0.045 does not mean an effect is "real," and a Bayes factor of 3.2 does not mean it is "likely." Both are continuous measures of evidence, and both should be interpreted in the context of prior knowledge, study quality, and effect size. The identical d = 0.20 from two pipelines is a reminder that numbers alone do not tell the full story; the pipeline that produced them matters just as much. The trade-off between frequentist and Bayesian approaches is not about which is correct, but about which is more appropriate for the question at hand. Frequentist methods are well-suited for controlling long-run error rates in confirmatory research, while Bayesian methods are better for updating beliefs in light of new data. Neither is inherently superior, and both have their place in a mature science.

In the end, the two labs reported the same effect size, but the scientific community will need to decide what to make of it—and that decision will depend as much on the methods as on the numbers. The replication crisis has taught us that no single study, however well-designed, can settle a scientific question. Instead, progress comes from accumulating evidence across multiple studies, using multiple methods, and being transparent about the choices that shape our conclusions. The two-lab comparison is a microcosm of this larger lesson: statistical pipelines are not neutral tools; they are lenses that focus and color what we see.