Each year, the federal government spends more than $30 billion on basic scientific research. Universities and private foundations spend around $20 billion more, according to one estimate.
Virtually none of that money is earmarked for research replication.
MATTERS OF FACT:
Exploring the Culture of Science.
At least from the outside, this can seem odd. Research replication, after all, is a time-honored aspect of science that requires experimental results to be tested and verified (or debunked) through repeat experiments, both by the original researchers behind those novel findings and, ideally, other researchers working independently. It’s often time-consuming, thankless, and expensive — but it’s also an essential part of empiricism, in that it gives the accumulation of scientific knowledge its durability and power.
Or at least that’s how it’s supposed to work. In reality, major scientific communities have been beset these last several years over inadequate replication, with some studies heralded as groundbreaking exerting their influence in the scientific literature — sometimes for years, and with thousands of citations — before anyone bothers to reproduce the experiments and discover that they don’t hold water. In fields ranging from cancer biology to social psychology, there’s mounting evidence that replication does not happen nearly enough. The term “replication crisis” is now well on its way to becoming a household phrase.
Suggested reasons for the crisis are many. Some researchers have blamed the scientific publishing culture itself, which privileges titillating new findings and discoveries, while providing comparatively little room for the publication of vital but less glamorous replication studies. But some stakeholders also point to a more concrete and obvious root cause: a lack of direct incentives to replicate other researchers’ work, including very little funding to do replications. This raises some key questions, including whether the federal government — the largest single funder of basic science in the world — ought to be doing more to encourage and underwrite basic replication research.
At the moment, only token reforms have been undertaken in the U.S., but there are signs that novel programs are gestating elsewhere, including a European pilot program that could serve as a model for replication funding in other countries.
To be clear, replication research does get funded. It’s just not very direct. Researchers often replicate their own work, of course, and they’ll replicate each other’s work, sometimes as the first step in doing other experiments. What’s lacking is the money and incentive structure needed to truly routinize replication. As it stands, there are often too few, if any, incentives — financial and otherwise — for researchers to spend a great deal of time re-testing their own studies, much less setting aside that work to take up the findings of other scientists and repeat their experiments for the greater good of the scientific process. Replication work isn’t flashy. It won’t get you tenure. Journals, especially high-profile journals, rarely want to publish a repeated version of a study. And of course, replicating experiments won’t necessarily make you many friends: Not all researchers want to hear that one of their studies didn’t hold up under scrutiny.
A survey of 1,500 scientists, conducted by the journal Nature last year, suggested that researchers often weren’t telling their colleagues — let alone publishing the results — when other researchers’ findings failed to replicate. (That survey also suggested that more than half of scientists consider the reproducibility situation to be a “crisis,” although perceptions vary from field to field.)
It’s not difficult to imagine a world where scientific funders set aside some proportion of their budgets to making sure that important new research actually holds up under scrutiny. “Before investing in a clinical trial, or just lots of dollars and new grants following up on it, let’s make sure that we nail down whether that initial finding is actually a finding to take seriously,” argued the psychologist Brian Nosek, a prominent replication advocate and executive director of the Center for Open Science, in an interview with Undark. “In that context, one or two percent of research budgets devoted to replication might be enough to really balance the efforts, the dollars, going to novelty, new areas of research, versus replications.”
What does that investment look like today? “It’s near zero,” Nosek estimated.
The United States’ two major scientific funding agencies, the National Science Foundation and the National Institutes of Health, have both made efforts to address research reliability, but neither currently sets aside money specifically for replication. Fay Lomax Cook, who runs the social sciences division at NSF, released a so-called Dear Colleague letter in 2015 inviting proposals for work that would “enhance the robustness and reliability of fundamental research,” including replications of foundational studies. In a phone interview, she told Undark that her division was distributing grants from the first round of funding — around $2 million — this year.
Of course, that $2 million is less than 1 percent of the annual budget for Cook’s division of NSF, and roughly 0.02 percent of NSF’s overall annual expenditure — though Cook emphasized that the effort was not just about offering money, but about sending a message. “If our scientists know that this is important, they will realize that when they submit proposals on something new and exciting and forward looking, that they might consider the first step toward that research as reproducing a finding that has gone before, so they can show us that they are standing of the shoulders of previous research,” she said.
She also emphasized that an attention to rigor shouldn’t be limited to a single set of grant awards. “Robust and reliable science … ” she stressed, “is integral to all of our research.”
Lawrence Tabak, the principal deputy director of NIH, told Undark that the agency was also placing more of a priority on replication work. “At NIH we have begun to see an increasing number of funding opportunity announcements that certainly mention that replication studies would be considered as being responsive to the funding opportunity announcement,” Tabak said. He also explained that the financial need wasn’t as great in the U.S. because of private donors. “In the United States, I think foundations have sort of stepped into this space, and there are a number of them that actually support replication studies.”
Perhaps that’s so, but in researching this column, I was only able to find one foundation in the United States that funds replication work. The Laura and John Arnold Foundation has underwritten influential replication initiatives in social science and cancer biology. (It is also the top funder for Nosek’s Center for Open Science.) But when I asked the foundation’s vice president of research, Stuart Buck, if they had enough money to cover the nation’s replication funding gap on their own, he sounded amused, pointing out that the NIH is a federal agency with a $30 billion budget. “I think we’ve probably put a couple of million into replication,” Buck said. “But $2 million, compared to $30 billion — that’s a difference of, what, 15,000?” If they dedicated even 1/1000th of their resources toward replication, that would be $30 million.
“I wish that the NIH and NSF would take some responsibility for some of this themselves,” Buck added, “because their annual budget combined is way more than what the Arnold Foundation has in total to spend — ever.”
A look abroad paints a somewhat more hopeful picture. In 2016, the Netherlands Organization for Scientific Research announced that it would give away 3 million euros (about $3.6 million) to scientists seeking to replicate foundational research. In July the agency, which goes by its Dutch acronym, NWO, announced that it was funding nine projects — although more than 80 different research teams had applied for funding (The chair of the grant program committee, Lex Bouter, communicating through NWO spokesperson Olivier Morot, declined to be interviewed without being permitted to review his quotes prior to publication.)
Asked whether the NIH would ever consider offering a similar grant, Tabak said that, while it’s “a reasonable model,” he’s not convinced the NIH needs to emulate it. Scientific communities, he argued, already tend to vet the work that’s most important. “Fundamentally, I believe in this self-correcting principle, that the importance of the finding kind of drives things. And I think that’s a good mechanism that we probably shouldn’t tinker with all that much,” he said.
Not everyone agrees that scientists themselves, prompted by only the barest of encouragement from their grant agencies, will fill the void. “This is not something that’s going to solve itself by just reminding people,” said Daniël Lakens, a replication advocate, a psychologist at Eindhoven University of Technology, and one of the creators of the Dutch grant program. “If you don’t force it in some way — this is like the least attractive thing for people to do by far,” he said in phone interview with Undark. “I mean, it’s incredibly risky for scientists to directly replicate the work of someone else.”
He also brought up the social pressures that can deter people, especially young researchers, who try to do direct, in-depth replications of their colleagues’ work. “Unless you have an extremely idealized view of what scientists do, and ignore the fact that they have a mortgage and want to keep their job, you can’t expect these things just to happen. I think that’s a very naive view of science,” he said.
Grant money alone can’t fix the problem, Lakens acknowledged, but it does offer one concrete incentive.
Curious to see what this looked like in practice, I called up Nynke van der Laan, a behavioral neuroscientist at the University of Amsterdam. Van der Laan received 90,000 euros (about $108,000) from the NWO to replicate — at long last — an influential 2009 Science paper that scanned the brains of people while they were making decisions about what to eat. The paper has been cited more than a thousand times, and it helped establish a new line of research into dieting, self-control, and the brain. But, while parts of its methodology have been replicated, nobody has ever actually redone the study, even though its sample size is smaller than the now-accepted standard for studies that make use of functional magnetic resonance imaging.
“I think a lot of people want to replicate [the study],” van der Laan said. “It’s just very expensive. That’s the difficult thing, of course. If you don’t have the money to do it, if you get a grant you’d rather use that for a new idea than to replicate it, because there’s more to win with a new study than with a replication study.”
The original study’s first author, Todd Hare, now a professor at the University of Zurich, told me that he’s excited to see the paper replicated — and said that the replication is years overdue. He does think his findings will hold up. But, he said, “I’m more excited to see this done in a really solid and unquestionable way.”
Van der Laan applied for the grant as a postdoc, and it will be one of the first projects she takes on as an assistant professor: a welcome bit of replication science will strengthen, or perhaps find weaknesses in, a highly influential study — close to a decade after the fact. Most critics would agree that’s just too long to wait.
In our conversation, she acknowledged that spending time on replication work might not be the best thing for her career, but she also said she was happy to have an NWO grant on her résumé — particularly given how competitive it was to win the funding. She also said that, as a scientist, she’s happy to do the work.
“I just want to do science that I think is important, and I think it’s really important to give an illustration of whether or not such a seminal study will replicate.”
Michael Schulson is an American freelance writer covering science, religion, technology, and ethics. His work has been published by Pacific Standard magazine, Aeon, New York magazine, and The Washington Post, among other outlets, and he writes the Matters of Fact and Tracker columns for Undark.