Recently, I received an email from a wildlife researcher — I’ll use the pseudonym Scientist A — who wanted to anonymously seek advice on a professional quandary. This researcher believes that two colleagues are presenting data on a controversial wildlife species in a misleading way. The dataset now spans roughly four decades, but when the method for counting the critters changed in the 1990s, the population census leapt by an unprecedented amount. While Scientist A says the switch in survey methodology created the illusion of a spike in the animal’s population growth, the colleagues — whom we’ll refer to as Scientists B and C — portrayed the increase as a biological reality. Scientist A further notes that although their older research reports clearly mentioned the shift to the newer census method, the more recent studies by Scientists B and C haven’t acknowledged it. That includes a new report this year that ignored other published work pointing out the importance of the change in methods. As a result, Scientist A is now pondering whether the right move is to write another commentary and continue discussing the issue publicly in the literature — or demand retraction of recent papers by Scientists B and C. “Is this a case for scientific debate,” Scientist A wondered in the email message, “or a case of misconduct?”
DILEMMAS:
Of science, ethics, and us.
Let’s start with the bad news for Scientist A: It’s impossible for us at Undark to really weigh in on the particulars of this conflict without knowing a great deal more detail from both sides. Scientist A views things one way, and Scientists B and C clearly view it differently — and we are in no position to adjudicate the findings of wildlife population science in this space.
That said, the more general issue confronting Scientist A — suspecting a misstep in the published work of colleagues and being uncertain of how to effectively deal with it — is not that unusual in the world of science. Errors that compromise the research literature, even just unintentional mistakes, continue to be a major problem, after all. And calls for more aggressive policing of peer-reviewed work have been on the rise for years, as have attempts to create safe-spaces for scientists to own up to their own research mistakes.
But at the heart of Scientist A’s lament are some fundamental questions that, sooner or later, every researcher is likely to run into: What’s the best way to address suspected inaccuracies or misleading information in the scientific literature, which is supposed to reflect our best approximations of reality? And how best to engage colleagues when there’s a dispute over data, attribution, methodology, or other aspects of presenting research findings?
There are no easy answers, but setting the record straight — and doing so in a transparent and professional manner — is all the more important when research results can shape policies that affect public health, the environment, or in this case, wildlife conservation and management. To get some perspectives on navigating this sort of predicament, I checked in with a few experts.
“We know that there’s a lot of inaccurate material out there,” said Nicholas Steneck, a science historian at the University of Michigan and consultant on research ethics and integrity. While that doesn’t mean that good science is not being done, he told me, “it certainly slows research, it wastes a heck of a lot of money, and we ought to do more to clean up the literature than we actually do.”
GOT A QUESTION FOR DILEMMAS?
If you’re a scientist or science journalist wrestling with an ethical dilemma of your own — or if you’re simply curious about an ethical quandary attending scientific research — send an email to [email protected]. We’ll do our best to find an answer. Anonymous queries are welcome.
As one case in point, Steneck recalled a postdoctoral researcher who told him that leading investigators in an emerging area of biomedical science had improperly calibrated some equipment. Consequently, most of the early studies were misleading, yet no one was willing to address it. “The leaders did not want to retract their earlier research and the postdoc was not willing to risk a career,” Steneck said. Dozens of the unreliable articles remain in the published literature, and while most experts in the field know that, outsiders may not have a clue.
Calling out others’ possible research blunders can be risky, and reluctance to do it is understandable. Those who wish to take action need to first take stock of how certain they are of their own conclusions, of course. But they also should consider how protected their own job position is, Steneck said. In the case of Scientist A, for instance, if Scientists B and C wield a lot of influence in their field, could criticizing their work hurt Scientist A’s ability to get funding for future projects? If so, that’s a real problem, said Steneck, who suggested that the wildlife researcher seek advice and support from a supervisor or department chair, as well as the scientist’s institution.
“When you do go into those situations,” said Steneck, “it’s best not to do it alone.”
While recent concerns about shoddy science have largely focused on problems in the areas of psychology and biomedicine, the field of ecology isn’t immune to the publish-or-perish pressures that encourage flawed and exaggerated research reports. Ecologists and evolutionary biologists commonly engage in so-called questionable research practices, according to a survey published last week by Hannah Fraser and Fiona Fidler of the University of Melbourne in Australia and Tim Parker of Whitman College in Walla Walla, Washington. Two-thirds of respondents admitted that they had, for instance, cherry-picked data on at least one occasion. (Such questionable research practices generally aren’t considered misconduct, though some people argue otherwise.)
When I briefly described Scientist A’s concerns about the wildlife population dataset to Parker, an ecologist and evolutionary biologist, he thought the situation sounded problematic. Most researchers would agree that studies should always report that a change in measurement methods was implemented at the very time-point when a big change in the measurements was observed, he said. Scientists are supposed to rigorously look at all the evidence bearing upon a question and evaluate it openly, so that others can assess whether their conclusions are right. “If you’re concealing some piece of information that undermines the case that you’re making, then clearly you’re not being transparent,” Parker said — and that, to him, potentially smells of misconduct.
However, by definition, scientific misconduct means that perpetrators acted intentionally to commit fraud, and that’s tough to prove.
What about pushing for retraction of the studies in question? If a publication is inaccurate — whether due to honest mistake, or misconduct — it should be retracted. Alternatively, if a study with otherwise sound conclusions nonetheless contains erroneous data, the paper should be corrected.
The trouble is, pursuing retractions or corrections is often an uphill struggle. Generally, going through all the “proper channels” for addressing flawed research — from writing letters to authors or journal editors to requesting a retraction — seldom works, said Ivan Oransky of Retraction Watch, a website dedicated to tracking retractions made by scientific journals.
While Retraction Watch doesn’t give advice on troubleshooting specific cases of questionable research, Oransky shared a few broad observations. “It may sound strange coming from someone who’s cofounder of Retraction Watch, but retractions are not always the best solution,” he said. If the goal is to clean up the scientific literature, demanding a retraction may be counterproductive because it instantly puts everyone “on edge,” he said. “If you immediately request retraction, because it’s got a stigma, because it’s got an association with misconduct and fraud, someone’s just going to get defensive.”
So where does that leave Scientist A? In cases like this, perhaps the best solution for dealing with inaccurate findings, Steneck said, is to “collect evidence and publish peer-reviewed articles that call the prior research into question.” He pointed to the example of Nathan Myhrvold, the wealthy former chief technologist at Microsoft with a physics Ph.D., who in 2016 disputed the accuracy of widely-cited asteroid data from NASA. Many asteroid experts didn’t take him seriously. But last month, Myhrvold got their attention by publishing his final analysis in a prestigious peer-reviewed journal.
Of course, few scientists have the resources to go to the lengths that Myhrvold did: He hired lawyers to file Freedom of Information Act (FOIA) requests to get NASA to fork over details of its asteroid analysis methods and algorithms.
Nonetheless, showing that there’s a solid scientific basis for questioning the reliability of published findings allows the research community to then evaluate the situation, Steneck said — and it’s an approach that doesn’t require criticizing others or claiming misconduct. Of course, when a field is too tightly controlled by thought leaders, Steneck added, it may be difficult to get dissenting ideas accepted by traditional top-tier journals, but you can probably still publish them in a peer-reviewed open-access journal. If Scientist A is in a position to pursue such a route, then over time, others in the field might come around to the researcher’s view of things.
Alternatively, in certain scenarios, it could make sense to try a less conventional approach: bringing the case to a “scientific sleuth” with a track record for shining light on faulty findings. As Oransky noted, some data detectives have had “a lot of success because they’re careful, they’re measured, they don’t accuse people of misconduct.” These research watchdogs show their analyses of the problem and just say, “This is a pretty serious error, I think the world should look at it,” he explained. They start off by writing polite letters to the authors of the questionable work, and their institutions, but if those efforts go nowhere, the sleuths then take their concerns public by blogging or going to journalists. The overall strategy can be “very powerful” in achieving retractions and corrections, Oransky said.
Otherwise, another way to call out suspect results is to use online post-publication peer review forums, such as PubPeer.com. But scientists who post comments there have to be meticulously careful, Oransky noted, “because it will backfire the minute you make a mistake and someone pounces on it.”
Hopefully, these perspectives provide some useful ideas for Scientist A in considering how to challenge the two colleagues. Frustration and pushback are likely no matter the approach, but a parting comment from Oransky seems worth bearing in mind: When you criticize other scientists’ work, he told me, they’ll probably react in the same ways that you would, if the roles were reversed.
Bottom line: Treat your colleagues as you would wish to be treated yourself.
Ingfei Chen is a California-based writer whose stories have appeared in publications including Scientific American, The New York Times, and Spectrum. She is a former Knight Science Journalism fellow at MIT.
Email us at [email protected] if you’d like to seek input on a quandary of your own. Undark will talk to experts on ethics, philosophy, or standards of ethical scientific or journalistic practice and share their best wisdom on possible solutions. For those wishing anonymity, we’ll withhold your name from any resulting Q&A items that we publish.
Comments are automatically closed one year after article publication. Archived comments are below.
I would like to thank Ingfei Chen for this article. ? Thank you.
I think this article indicates pretty clearly that the scientific process has broken down in regards to errors and/or misconduct. We scientists always talk about the open debate required of science but for a variety of reasons, that debate is not taking place when it comes to this issue. Yes I recognize the factors that keep scientists from addressing the issue here and others like it (and were noted in the article, like career suicide for a post-doc for example). But the process is not working as it should. It would seem the open debate part of scientific research only applies to competing studies that were done correctly but come to different conclusions. Open debate is very nearly non-existent for situations for error and/or misconduct. I agree with the article that going right to misconduct charge is not the way to go initially at least. But there should be debate about suspect errors. An open discussion of such errors would bring others with appropriate expertise into the debate and flesh out whether something is in error. After that has occurred institutions or scientific societies can examine whether it was due to misconduct. And I know that most institutions and societies right now would likely not fulfill this role unless the conduct was sufficiently egregious to harm their program or reputations. Institutions themselves are part of the problem as they have “invested” in the people working for them and thus are not unbiased actors in this situation. But nevertheless they should be playing this role, another break down in the research process. While this article is very good, I find scientists too willing to justify keeping things as they are. Possibly losing a grant is not a legitimate reason to not perform the role that has been entrusted to scientists. This article and others I have read on this topic seem to give short shrift to the ethical debate. Yes I know there is a lot of competition for grants, thus to remain competitive scientists are pressured to publish a lot and frequently. So what? This is not justification for allowing the issues of errors and/or misconduct to go on. It is simply an excuse to not do something that is hard, but is front and center of the scientific process, debating the science. No you don’t just debate when conflicting studies are competently done, you need to debate when there are errors too. That is rarely happening. If the obstacles to doing this, such as career suicide are too big to overcome, then we need an anonymous forum where this debate can take place. It appears baby steps are being made with organizations like Retraction Watch playing some role, but this issue is scientists themselves corrupting the scientific process and should not require a whole new organization to get scientists to do what they should be doing. OK, you want to do something anonymously, fine. Scientists themselves should be pushing hard to set up a credible process where that debate can take place. But this is not what is happening. Scientists are shirking their duty and responsibility entrusted to them by not doing so. Perhaps on of the journal publishers could do this and set it up in a way where there is a continuing debate about any questionable results, all sides can have their input and probably would need moderation so that it is used by verified scientists. Opening it to the public would allow to much social media “noise” as can be seen on any internet discussion forum. And likely just the fact that something is being discussed would get those publishing questionable results to lay out their data, explain their justification given how scientists care so much of their own reputation. The results methodology could be delved into with greater detail, with back and forth among those taking one side or another. This by itself would be a huge step forward. A step farther: allow discussion to take place for a reasonable period, then have anonymous peer reviewers examine the discussion (and be part of the discussion as well mind you), and like all journal articles put forth judgement. They may determine that an error in methodology has occurred and recommend a correction. This recommendation would have no force in that this organization could not enforce their recommendations, only suggest them. Scientists publishing the papers with errors should be allowed to correct the papers as an addendum to the title, abstract and a new section. For example, the original paper and its content would remain, the addendum might read that such and such an error occurred and the findings of part or all of this paper are incorrect with explanation. Why keep the paper rather than retract? Over time this information may have become part of the “institutional memory” of findings on the topic. Removing the paper or simply retractiing may perpetuate the errors carried forward due to that institutional memory. Correcting the paper helps stop the continued use of that information as there is now a publication record showing this finding or part of a finding are no longer valid, and it is right there, documented in the journal. If you are a scientist worth your salt, you will keep up with this and immediately correct that institutional memory for yourself. Instead what we have now are disagreeing scientists quietly talking among themselves, and not the publishing authors, and slowly, people working in the field become aware of a potential error. A very very inefficient process also not accessible to all who have not interacted with this group of “informal in the know” scientists. In addition to this process, we need a journal for publishing negative results. Negative results are useful information and also would allow scientists to publish their findings that directly contradict earlier work, which itself would foster more discussion on the issue at hand. The scientific community also culturally needs to support published negative results as a quality study that is as important as studies that find positive results. This is needed so that scientists take the time to publish negative results as it is worth their time to do so. These negative results don’t need to necessarily suggest error on the part of other studies with questionable findings, although ideally they would point such things out. But as we have seen scientists have many excuses for not doing just that. So simply publishing the non-confirmation would suffice for drawing attention to questionable data. I too have experienced dealing with results from others I could not duplicate and wasted a year and a half of a post-doc to figure it out. That is a lot of time to waste because scientists don’t want to criticize other’s results for fear of (insert excuse here). This is a big problem and a lot of time and resources are being wasted as a consequence, and even damaging the career of others who rely on these types of results. We are supposed to debate results in open discussion, this part of the scientific process has broken down for questionable results.