A Prescription for Better Science

What’s wrong with science? Stanford University’s John Ioannidis has been asking this question for a long time – at least since his 2005 article, “Why Most Published Research Findings Are False.” And now, 12 years later, he and a slate of co-authors from the United States, the U.K., and the Netherlands, have some suggestions about how to solve the problem.

In a new paper, a group of researchers propose the “adoption of measures to optimize key elements of the scientific process.” Will it work?

The paper, titled “A Manifesto for Reproducible Science” and published this month in the journal Nature Human Behavior, makes the case that what’s wrong with science is simple: It’s us. As humans, we’re ill-equipped to see the world as it is; the world as we’d like it to be is much more compelling to our cognitive decision-making apparatus. We look for results that confirm our biases and we’re hard-wired to see causality where there is only randomness.

This is not a recipe for effective truth-seeking.

If scientists (and their partners in truth-seeking, journalists) could wave a magic wand to erase their biases, they all would have done it long ago; I know I sure would have. But none of us can do that, and science can’t expect it of us. Instead, it has to correct for our biases. “Science” does not mean “truth.” Science is a process by which we hope, collaboratively and over time, to arrive at the truth.

If most published research findings are false, science is clearly not firing on all cylinders. So what can be done about that?

A lot, say the authors — though what strikes me as most notable about their manifesto is that it is less for individual scientists than for science itself. It provides prescriptions for bringing more robust checks and to balances to published science, to achieve collectively what no single scientist — nor even the hoary process of peer review — can possibly accomplish.

Among their recommendations (along with my interpretations):

Registration: Defining study methods, goals, outcomes, and statistical techniques ahead of time establishes goalposts, and allows other scientists to see whether the published work strayed from the original plan. If it did, they can ask why. The majority of clinical trials are already registered, but other research seldom is.

Blinding: Everyone involved should be kept in the dark about which experimental groups generated which data and even, where possible, the study hypothesis. While the importance of blinding is Science 101, the authors estimate that it’s used in only 5 percent to 30 percent of published papers.

Open peer review:  Peer review has long been a staple of academic publishing, but the internet offers ways to improve it. There are already services that allow scientists to disseminate papers, data included, before publication to get feedback from the research community.  While conventional peer-review is anonymous, and limited to just a few reviewers, open peer review means anyone – including people with opposing ideas or worldviews – can comment, for all the world to see.  According to the paper, “there is evidence that open peer review improves the quality of reviews received.”

Disclosure and mitigation of conflicts of interest: As with other fields, scientists must navigate a variety of personal biases and conflicts, financial and otherwise. These include “beliefs and preconceptions of individual scientists and the stakes that researchers have in obtaining publishable results in order to progress.” While conflicts of interest are already widely disclosed, strategies to mitigate those conflicts (such as including independent researchers in the process) are rare.

Data sharing: When scientists release data, other scientists can verify and build on results.  Without access to data, it’s impossible to evaluate the statistical analysis – and that’s one of areas in which motivated reasoning rears its head.  When statistics software offers a smorgasbord of analytical choices, “biases can encourage the acceptance of outcomes that fit expectations,” the researchers note. And so we end up with the problem of p-hacking: the parsing of results in different ways until a statistically significant – and congenial – result emerges. Giving other scientists, whose idea of congenial might be different, access to the data can provide a check – in both senses of the word.

Team science: A key problem is “scientists working competitively in silos without combining their efforts.” Collaborating with other groups means different ideas, different experiences, and different perspectives all come into play, reducing the risk of bias.

A list of concrete ways to improve the scientific process has the added benefit of giving science-minded journalists and readers some help answering the question: How good is this paper? Ivan Oransky, co-founder of the science watchdog site Retraction Watch (“tracking retractions as a window into the scientific process”), told me the paper is “is a wonderful distillation of a large number of very important issues that researchers should all be thinking about.”

But he also warns against using the suggestions as a be-all-end-all checklist for good science. You can have a lousy study that checks a lot of the boxes, Oransky notes, adding that “no checklist is a substitute for actually reading the study.”

Still, the boxes are important, and making the issues they represent part of the public conversation about science might, in the long run, help defuse some of the more contentious scientific issues playing out in the public sphere. If scientists can collaborate, share data, and engage with critics, maybe there’s hope for the rest of us.

Tamar Haspel is a journalist who has been writing about food and science for nearly two decades. She writes the James Beard Award-winning Washington Post column Unearthed, which covers food supply issues, and contributes to National Geographic, Fortune, and Cooking Light.

CORRECTION: This post has been updated to accurately state that 2005 was 12 years ago, not 17.