David Randall and Christopher Welser are unlikely authorities on the reproducibility crisis in science. Randall, a historian and librarian, is the director of research at the National Association of Scholars, a small higher education advocacy group. Welser teaches Latin at a Christian college in Minnesota. Neither has published anything on replication or reproducibility.
But when a report the two men wrote, “The Irreproducibility Crisis of Modern Science,” was published by the National Association of Scholars on Tuesday afternoon, it received a Congressional reception. The launch took place in a House office building on Capitol Hill. The Texas Republican Lamar Smith, chairman of the House science committee and one of the most powerful science policymakers in Washington, spoke at the event. In a statement to Undark, he described the NAS report as an “important study.”
The report offers a lucid overview of the reproducibility debate. It also suggests 40 measures to help scientists produce more rigorous, reliable research. Most of these proposed reforms will sound familiar — and welcome — to scientists concerned about the issue.
Other details of the report, though, promise to raise concerns — starting with the involvement of Smith, who has dismissed climate science — or what he calls “the climate change religion” — as liberal alarmism. And indeed, the National Association of Scholars report alludes repeatedly, without offering much evidence, to a looming reproducibility crisis in climate science itself. The organization has a history of promoting climate denialism, and William Happer, a Princeton physicist and climate change skeptic who argues that extra carbon dioxide is good for the planet, wrote the report’s afterword, in which he accuses scientists of hoodwinking “deplorables.”
The report also calls for legislation, long championed by Smith, that many scientists argue could suppress research and helps Republicans in Congress roll back environmental regulations.
To be sure, reproducibility issues pose serious challenges for scientific communities. But what happens when those issues get picked up by political activists? And what, exactly, does the National Association of Scholars hope to achieve with the report?
The NAS (its acronym, confusingly, mirrors that of the National Academy of Sciences) was founded in the 1980s, with a mission to defend the values of the liberal arts. Its membership draws heavily from the ranks of conservative academics. “Pretty much from the beginning, we took threats to the integrity of good science as one of our topics,” NAS president Peter Wood told me. “Anyone who’s concerned about the health and welfare of sciences needs to be concerned about the amount of science that doesn’t stand up to scrutiny,” he added. “Just because that’s part of who we are and what we do, it seemed like it was about time to try our hand once again at a more systematic treatment of the problem.”
Randall, the report’s co-author, told me that reading a profile of John Ioannidis in The Atlantic had first gotten him interested in reproducibility issues. Ioannidis, now a physician and researcher at Stanford, wrote a blockbuster paper in 2005 expressing “increasing concern that most current published research findings are false.” In the years since, Ioannidis has become a prominent science reformer, and his colleagues have debated whether his claim — if it’s correct — reflects the natural, erratic progress of science, a deep flaw in the modern scientific enterprise, or something in between.
Certainly, the work of Ioannidis and others has helped draw attention to the way scientists review and publish each other’s research. And the NAS report echoes many suggestions that reproducibility advocates have made over the years, including raising the threshold for statistical significance, increasing funding for research replication, pushing researchers to share data, and setting standards that require journals to be more transparent about their process of peer review.
“There is a lot to like in the report,” Brian Nosek, executive director of the Center for Open Science and a prominent reform advocate, wrote in an email to Undark. Andrew Gelman, a statistics professor at Columbia University and a vocal critic of what he characterizes as lax standards in social psychology and other fields, agreed. “Overall, I was happy with the report,” he said, although he admitted that he had not read it closely. (Randall described Gelman to me as “a model public intellectual” and cited him as an inspiration for their work).
Randall and Welser insist that their goal is to help depoliticize science, by pushing for more objective, reliable research. And yet there is a clearly discernible political edge to their work — and that’s likely in part because of NAS’s own history with science and climate issues. The organization regularly publishes articles assailing mainstream climate science. Wood, an anthropologist by training, has written about “the bogus ‘global warming consensus,’” and in a Wall Street Journal op-ed published Monday and linked to the report, he and Randall describe “the whole discipline of climate science” as “a farrago of unreliable statistics, arbitrary research techniques, and politicized groupthink.”
The organization has received donations from the Charles Koch Foundation, one of the country’s most prominent anti-climate funders. It receives a sizable portion of its annual budget from the Sarah Scaife Foundation, which climate change activists have identified as a major backer of attacks on climate science.
Kerry Emanuel, a climate scientist at MIT and former member of NAS, says he was initially drawn to the organization because he was worried about what he saw as a growing relativism in the academy, evident in the work of deconstructionist philosophers like Jacques Derrida. NAS seemed to be taking a stand against those intellectual currents, Emanuel said — though he adds that he eventually became concerned about the organization’s stances on climate change, especially during a much-publicized incident in which hackers stole thousands of emails from a group of climate scientists and accused them of misusing data.
In a 2010 article published on the NAS website, Emanuel described the event as “a scandal” — but he didn’t see it as a challenge to the scientific consensus on climate change. The National Association of Scholars, on the other hand, sought to extrapolate the Climategate incident “into a universal condemnation of the field,” Emanuel told me. “It was just patently disingenuous.”
He left the organization soon afterward.
“It sort of revealed them not to be what they claimed to be — people who stood for scientific truth and scientific integrity. It was just another organization that used that as a front,” Emanuel said. “They’re basically a political organization posing as an organization dedicated to free inquiry,” he added.
Wood disputed Emanuel’s characterization of NAS. “It’s a false charge,” he said. “Professor Emanuel is someone who seems to have a closed mind on the issue of climate change.”
For Emanuel, though, the organization’s stance on climate wasn’t just misguided — it was a betrayal of their mission to challenge the deconstruction of truth. “They were publishing articles that were just as bad as the ones that the organization was founded to counter,” he said.
For many scientists and advocates, the most concerning part of the National Association for Scholars’ report is likely to be its advocacy for the Secret Science Reform Act — a piece of legislation, first introduced in 2015, that would bar the Environmental Protection Agency from using research that is not “substantially reproducible.” Lamar Smith has been the chief sponsor of the bill, which was recently renamed the HONEST Act. Scott Pruitt, the Trump administration’s embattled EPA director, has tried to apply some of its principles at the agency.
Supporters of the bill, including the NAS authors, argue that the legislation makes sure that policymakers only use rigorous research. The NAS report suggests expanding the policy to all federal agencies, and to federal courts. But major scientific organizations have been opposing the Secret Science Reform Act for years. Sean Gallagher, a senior government relations officer at the American Association for the Advancement of Science, expressed concern that the term “reproducible” in the act is so vague that it could be used to freeze good research out of policy discussions. “Reproducible means different things to different disciplines,” Gallagher told me.
Smith and others, Gallagher said, have done little to heed the concerns of AAAS and other organizations. Stressing that he had not yet reviewed the NAS report, Gallagher suggested that efforts like the HONEST Act might not be about upholding rigorous scientific integrity standards. “You do get the sense that they don’t like some of the regulations that used science, and therefore they attack the science.”
Nosek expressed similar concerns in his email to Undark about the NAS report. “Restricting policymakers to use only evidence that meets the highest transparency and reproducibility standards would prevent policymakers from using the best available evidence in many different situations,” he wrote.
Welser and Randall said they were sensitive to scientists’ concerns. Their report, Randall said, “is a framework for further conversation and debate, which absolutely must involve scientists and policymakers together.”
Perhaps more than anything, the National Association of Scholars — and the ensuing backlash — serve to underscore some fundamental questions that linger at the center of the reproducibility debate: What does serious, constructive criticism of the scientific enterprise look like? And when does that criticism tip over into unwarranted skepticism — or even feed ideologically motivated attacks on researchers’ work?
Naomi Oreskes, co-author of “Merchants of Doubt,” an influential study of foundation- and industry-backed attacks on science, said the reproducibility debate has already been exploited by political activists. “These guys are loving it,” Oreskes, a historian of science at Harvard, told me. “Any time scientists themselves admit there’s a mistake or a problem, they’re all over it. They have a feeding frenzy because this is exactly what they want. And what they want to do is use this now to try to discredit all science.”
Just as concerning as ideological attacks, though, is the possibility that scientists will be hesitant to speak openly about the messiness of the work that they do. It’s striking that, even in an era of skepticism and conspiracy theories, scientific communities have still been able to have such open — and often contentious — debates about reproducibility.
“Ideology is a powerful influence over all our behavior,” Nosek said. “The best that we can hope for is exposing it with transparency, and correcting for it with open, constructive debate.”
Michael Schulson is an American freelance writer covering science, religion, technology, and ethics. His work has been published by Pacific Standard magazine, Aeon, New York magazine, and The Washington Post, among other outlets, and he writes the Matters of Fact and Tracker columns for Undark.
I worry about the conspiracy theories that seem to be getting more traction in the USA. This type of thinking muddles what a fact even is.
And by debating this nonsense as if it has any credibility, we fail to deal with the big issues of the 21st century. Good luck discussing anything with a conspiracy nut job who is convinced of whatever. They will stick to their guns and double down on their opinions no matter how outlandish and obviously mistaken their theory is. The moon landings were faked. September 11th was planned by President Bush. The Sandy Hook shooting didn’t happen, it is just a fake story to take away our guns, We have our former USA president still being accused of being secretly born in another country. Oh and I live in California, and apparently all the illegal aliens vote, and not only that, they vote multiple times. Who knew? Both sides of the political spectrum are full of crazy conspiracy theories. I just find the ones I listed to be the most offensive and dangerous.
Some people even think the earth is flat and the center of the universe. Wow!! Global climate change seems easy to pick on that if you accept the other BS. Science is hard, and a good scientist or mathematician double and triple checks their work. Science is messy. I know this because I went to a University and studied math and science. When I did this in the late 80s and early 90s, it was a good thing. Now, apparently I am a puppet of the deep state, whatever that is. If we don’t get off this crazy train we are screwed as a society.
Confusion and partisanship muddles our ability as a society to plan for the future. If we want to have a discussion of global climate change, or sea level rise as the military likes to call it, it’s rate, our strategy as a society, that would be great. Heck, the Koch brothers, famous conservatives, assist in sponsoring Nova, a science show on PBS, that does just that. But most people seem to want to cling to whatever they know and go into an social media echo chamber, so they don’t have to think and question.
What about AI in weaponry and self driving cars? Ethics pertaining to cloning or bioengineering? How about dealing with rising population; over 7 billion and rising? Balancing the needs of society with protecting the environment so our children and grand children don’t inherit a mess? What about nuclear power? Space travel and exploration, I hope?
Discussion of science and its implications from society seems more about shouting from both ends of the political spectrum. Which has a cooling effect on scientists frankly discussing problems that are the foundation of the scientific method. The rabid pundits from both ends of the spectrum are so eager to catch the scientists in a lie, or mistake any disclaimer of experimental error and misinterpret it into whatever conspiracy they are trying to “prove”, that real science is being buried.
Does the ‘Henry Miller’ persona understand molecular spectroscopy? Certainly not.
How is the public supposed to trust science that does not follow the scientific method? Consensus is mentioned. It has nothing to do with science. The history of science is replete with examples of consensus breaking. Reproducibility, which is truly in need of some revision, is also necessary as a check against confirmation bias. Science paid for by the public must be open to the public, with national security exceptions. The article seems to take umbrage at attacks on so-called global warming science. However, The IPCC and other pushers of global warming alarmism are so infested with politics and reliance on unreliable model predictions that any hint of scientific integrity has been lost.
It appears that Mr Berg is advocating hiding the evidence.
So apparently you’re all for reproducibility–except when it’s demanded of the AGW “religion.”
For at least a couple of decades, AGW “researchers”–and I use the term loosely–have, instead of observing climate in order to understand its mechanisms, been observing climate in order to confirm their pre-existing conclusion that humans are screwing it up.
This isn’t science, and the irreproducibility of the climate evolutions shown by the many various climate models demonstrates that. Look at the various previously predicted future temperature profiles and compare them to actual measurements. The models, almost universally, predict higher than measured. Wouldn’t you think that they’d spread around reality? That they don’t, that they’re near-universally high suggests either that the “scientists” involved haven’t a clue what they’re talking about–or that the models reflect their biases more than they reflect reality. The only “reproducibility” here is that, one way or the other, the “science” is bad.
As a cautionary word, if you want to write about reproducibility, good onya. As science evolves, things are likely to get more and more subtle and reproducibility will get more and more difficult as the number of “unknown unknowns” increases in proportion to the “known unknowns.” It’s not always obvious what’s relevant and what isn’t. But if you’re going to write about reproducibility, it might be best to avoid editorialising about contentious issues like AGW. It has become a religion, and no one ever wins religious arguments.
Yes! I agree completely… universities are one of the foundations of the Deep State pyramid. We need more organizations like NAS to MAGA!!!!
It’s funny that you cite political incorrectness to deflect the charge that much university science involves fraud. The issue that the report discusses isn’t a specific position on climate change but irreproducibility. The report discusses issues like fraud, small sample sizes, overemphasis of small (but possibly significant) effects (which may be statistical artifacts or due to fishing). Instead, you seem to emphasize the funding sources of NAS, and you aim to smear them by characterizing them as “small” and an “advocacy group.” Let me clue you in: All of American universities are funded by corrupt banking interests, starting with the Carnegie Foundation for the Advancement of Teaching and the Rockefeller-funded General Education Board. Your picking on the Koch Foundation is one more example of tendentious bigotry. Why doesn’t the University of Chicago hand back its endowment to the Rockefeller family and admit its been on the payroll for the past 125 years? Or don’t you know?
It needs to be noted that NAS not only provided Emanuel with a forum for his complaint against the manner in which Randall and Welser wrote about Climate Gate but also published his letter in their periodical Academic Questions.
Readers may be interested in this recent editorial from Jeremy Berg, editor-in-chief of Science journals: “Transparency is critical when it comes to decision-making that broadly affects the public, particularly when it comes to policies purported to be grounded in scientific evidence… The scientific enterprise should stand up against efforts that distort initiatives aimed to improve scientific practice, just to pursue other agendas.” http://science.sciencemag.org/content/360/6385/133.full