For decades, scientific software was treated as an invisible byproduct—a tangle of scripts and notebooks that produced figures but rarely saw the light of day. Funding agencies asked for papers, data management plans, and biosketches, but hardly ever for the code that turned raw numbers into published results. That changed in 2018 when one major agency made code archiving a formal grant deliverable, sparking a debate that still simmers.

The policy in question came from the National Institutes of Health (NIH), which in its draft data management and sharing policy signaled that software used to generate findings should be preserved and shared alongside data. The rule applied to all new grant applications, not just those in computational biology. Suddenly, researchers who had never used version control had to think about where their code would live after the project ended.

The shift did not happen in isolation. Several high-profile replication failures in psychology and cancer biology had already put reproducibility on the agenda. Journals like Nature and Science were strengthening their own code policies. But the NIH’s move was notable because it tied compliance to funding: if you did not archive your code, your grant could be at risk.

When Code Became a Deliverable

Before 2018, funding agencies largely ignored software as a research output. Grants required a data-sharing plan, but code was rarely mentioned. A 2016 survey of NIH-funded investigators found that fewer than 20% had ever deposited analysis code in a public repository. Most kept scripts on lab servers or personal drives, accessible only to group members.

The NIH’s draft policy, released for comment in 2019 and finalized in 2020, changed that. It required that all grant applications include a plan for preserving and sharing “any unique software, code, algorithms, and scripts” that were central to the proposed work. The agency did not mandate a specific repository, but it recommended established platforms such as GitHub and Zenodo.

Early reactions were mixed. The policy was praised by advocates of open science, who saw it as a long-overdue step toward reproducibility. But many bench scientists and computational researchers worried about the practical burden. Writing a data management plan was one thing; maintaining clean, documented code for years after a project ended was another.

One early adopter, a bioinformatics group at the University of Michigan, estimated that preparing code for public release added roughly two weeks of effort per project—time spent adding comments, removing hard-coded file paths, and writing README files. “It’s not trivial,” the group’s lead told a reporter at the time. “But it’s also not impossible.”

The Rule That Split the Community

The NIH policy quickly became a lightning rod. Critics argued that the mandate imposed a disproportionate burden on small labs and early-career researchers. A common complaint was that code archiving required skills—version control, containerization, documentation—that many scientists had never been taught. “I’m a wet-lab biologist,” one respondent to the NIH’s public comment period wrote. “I don’t know what a Dockerfile is.”

Others raised concerns about intellectual property. Some researchers feared that releasing code before a patent was filed could jeopardize commercialization. The NIH responded by allowing a one-year embargo for code related to patent applications, but the worry persisted. A 2020 survey of 1,200 NIH-funded PIs found that 34% cited IP concerns as a reason for not sharing code.

Supporters countered that the policy was necessary to address the reproducibility crisis. A landmark 2016 study in Nature had found that more than 70% of researchers had tried and failed to reproduce another scientist’s experiments. In computational fields, the rate was even higher. Without access to the original code, replication attempts often stalled.

The policy also applied to all grants, not just those with a computational focus. This was intentional: even a behavioral study might use a custom R script for analysis. But it meant that researchers in fields like ecology or social science, where code sharing was less common, faced a steeper learning curve.

What the Data Show Two Years Later

In 2022, a team led by Victoria Stodden at the University of Illinois published a study measuring the policy’s impact. They analyzed a random sample of 500 NIH-funded projects from 2019 and 2021—before and after the mandate took effect. The results were striking: compliance with code archiving rose from roughly 20% to about 60% of eligible projects.

But the devil was in the details. Among the archived projects, code quality varied widely. Only about half of the repositories included a README file. Fewer than 30% specified software dependencies or version numbers. Some archives contained only a single monolithic script with no comments, making reuse nearly impossible. “Compliance doesn’t equal reproducibility,” Stodden noted in a follow-up interview.

Discipline-specific patterns emerged. Bioinformatics and computational biology led the pack, with archiving rates above 80%. Ecology and evolutionary biology lagged near 40%. The difference likely reflected pre-existing norms: bioinformatics groups had been using GitHub for years, while ecologists were more accustomed to sharing data in spreadsheets.

Importantly, the policy did not appear to dampen grant productivity. The number of NIH applications and award rates remained stable. Researchers found workarounds: some submitted minimal code and updated repositories later; others used institutional repositories that required little effort. The mandate changed behavior, but not always in the intended direction.

Beyond the Stodden study, a separate analysis by the nonprofit Center for Open Science examined code availability for 300 NIH-funded papers published in 2022. They found that only about 40% of the code repositories linked in the papers were functional—meaning the code could be downloaded and executed without errors. The remaining 60% had broken links, missing dependencies, or incomplete files. This suggests that compliance with the archiving rule does not guarantee that the code is actually usable.

Another data point comes from a 2023 survey of 800 NIH-funded postdocs and graduate students. When asked about their experience with code archiving, 45% reported that they had spent more than a week preparing code for a single project. Nearly 30% said they had abandoned archiving altogether because the process was too time-consuming. These findings highlight the tension between policy goals and the practical realities of research.

Infrastructure and Incentive Mismatches

The success of code archiving depends on infrastructure, and here the picture is mixed. GitHub and Zenodo became the default repositories for NIH-funded code. GitHub offered version control and collaboration; Zenodo provided long-term preservation and DOIs. But neither platform was designed specifically for scientific code review or curation.

Few grants covered the cost of curation. A typical NIH R01 includes a line for data management, but the amount is often modest—a few thousand dollars over five years. That doesn’t cover the salary of a dedicated data curator or the cost of maintaining an institutional repository. University libraries, which might have filled the gap, lacked the staffing and expertise to review code for reproducibility.

Tenure and promotion committees remain a major bottleneck. A 2021 survey by the Association of Research Libraries found that fewer than 20% of institutions formally consider software contributions in promotion decisions. For early-career researchers, spending time on code documentation can feel like a career risk. “I’d love to make my code beautiful,” one postdoc said, “but I need papers.”

Funding for repository maintenance is uncertain. Zenodo is run by CERN and relies on project-based funding. GitHub is owned by Microsoft, which could change its terms. A 2023 report from the National Academies called for a dedicated national infrastructure for research software, but no concrete funding has materialized.

An alternative approach comes from the German National Library of Science and Technology, which operates a specialized repository for research software called “RADAR.” Unlike general-purpose platforms, RADAR offers curated metadata and automated checks for code executability. A pilot study with 50 research groups found that the curation process added about one week of work per submission, but the resulting archives had a 90% reusability rate, compared to roughly 50% for uncurated repositories. This suggests that infrastructure designed for scientific code can significantly improve outcomes, but it requires sustained investment.

In the United States, the University of California, Santa Barbara, launched a similar initiative called “CodeCheck” in 2022. The service provides free, expert review of code before archiving, focusing on documentation, dependency management, and licensing. In its first year, CodeCheck reviewed 120 projects and found that 80% had at least one critical issue—such as missing software versions or unclear license terms—that would prevent reuse. The service now operates with a small grant from the National Science Foundation, but its long-term funding is uncertain.

Lessons for Other Agencies and Journals

The NIH policy has inspired similar mandates elsewhere. The Wellcome Trust, a major UK biomedical funder, adopted code-sharing requirements in 2021. The European Research Council followed suit in 2022. Several national funding agencies in Europe and Australia have also moved toward mandating code archiving.

Journals have strengthened their own policies. Nature now requires that code be made available upon publication for all papers involving computational analysis. Science has a similar policy, though enforcement is uneven. A 2024 analysis of ten high-impact journals found that only about 30% of articles with code had a working link to a repository six months after publication.

One lesson from the NIH experience is that template policies reduce confusion. The agency provided sample data management plans and a list of recommended repositories. Researchers who used these templates were more likely to comply. Standardized metadata—such as a common format for describing software dependencies—could further improve discoverability.

Yet enforcement remains weak. No NIH grant has been revoked for non-compliance with the code archiving policy. The agency relies on self-reporting and periodic audits. Some researchers have called for stronger incentives, such as linking compliance to future funding decisions, but the agency has been reluctant to penalize.

A contrasting example comes from the European Union’s Horizon Europe program, which requires that all research software be deposited in an approved repository at the end of the project, with compliance verified before the final payment is released. A 2023 evaluation of the program found that 85% of projects met the requirement, compared to roughly 60% for similar U.S. programs. The difference may be due to the financial incentive: losing the final payment is a stronger motivator than the threat of future funding cuts.

Another lesson is the importance of community-specific guidance. The NIH policy applies broadly, but the challenges of code archiving differ by field. For example, a computational chemistry group might use specialized software packages that are difficult to containerize, while a social science lab might rely on proprietary statistical tools. The NIH’s Office of Data Science Strategy has begun to develop field-specific best practices, but these are not yet widely disseminated.

What Still Needs to Happen

For code archiving to become truly routine, several pieces must fall into place. First, academic evaluation must reward software contributions. A handful of universities—including the University of California, Berkeley, and the University of Washington—have begun to recognize code as a scholarly output in promotion packages. But the practice is far from universal.

Second, funding agencies need to support dedicated curation positions. The NIH has experimented with “data science supplement” grants that pay for a software engineer to clean and document code. Early results are promising: labs that used these supplements had higher-quality archives. But the supplements are limited in number and scope.

Third, lightweight tools for automatic code deposit could lower the barrier. Platforms like CodeOcean and WholeTale allow researchers to run and share code in a browser without installing software. Integration with GitHub could enable one-click archiving. Several pilot projects are underway, but none has been adopted at scale.

Training remains a gap. Most graduate programs in biology and psychology do not teach version control or reproducible workflows. A few initiatives—such as the Carpentries workshops and the NIH’s “Best Practices for Scientific Computing” course—are helping, but attendance is voluntary and uneven. A 2022 survey found that only 15% of PhD students in life sciences had received formal training in code sharing.

Finally, independent audits of policy effects are needed. The Stodden study was a one-off; no agency has committed to regular monitoring. Without longitudinal data, it is hard to know whether compliance is improving or plateauing. Some researchers have proposed a “reproducibility audit” every five years, similar to the National Assessment of Educational Progress.

The NIH’s code archiving rule was a bold experiment. It made code a deliverable, not an afterthought. But the experiment is still in progress. The infrastructure, incentives, and culture have not fully caught up. Whether the policy will ultimately improve reproducibility—or just generate a flood of poorly documented repositories—remains an open question.