Tweaking Drug Trial Design in the Hunt for the FDA’s Green Light
Starting in 2009, James Simon, a gynecologist from Washington, D.C., was trying to figure out how to tell whether a drug called flibanserin — which some later incorrectly dubbed “female Viagra” — actually did the job. Flibanserin had been developed as an antidepressant, but the pharmaceutical companies behind it believed it might have promise as a libido enhancer for women. Simon was one of the researchers hired by the drug makers to find evidence that would convince the U.S. Food and Drug Administration to approve it for that use.
Earlier studies on flibanserin focused on use by premenopausal women. Simon’s study would determine whether the drug was also effective for postmenopausal women as well.
Simon decided to have women in his test study self-report any libido changes, using an electronic diary. Self-reports are notoriously subjective, he conceded in an interview, and critics say they’re not useful for developing reliable statistics. Yet, Simon noted, the FDA approved the methodology in advance. According to Simon, when it became clear that the research approach was indeed flawed, the sponsor, Boehringer Ingelheim changed their data collection process part way through the study. This, experts say, ran the risk of making the results even less reliable.
FDA documents suggest that these decisions were possibly made by the drug companies themselves in an effort to paint flibanserin’s effects in a more positive light.
Everyone involved with the drug approval process — from drug makers to regulators to reform advocates — acknowledges that the incentive to fudge things is strong. A drug company wants to get its drugs to market so it can generate profits. The FDA regulatory requirements stand in the way — and are supposed to act as a check on proper clinical trial design. “You’ve got this FDA with its big thumb on what you have to do,” Simon said. “Go too far outside that, and you get smacked in the face.”
But in the end, the companies backing flibanserin and the scientists conducting the trials didn’t get smacked in the face, despite repeated modifications of trial designs. Of the 16 clinical experiments that were initiated on flibanserin — some of which are still underway — the majority of them had changes to their protocols. This means that each of these experiments differed from what was originally approved for implementation by the FDA. For 11 of the 16 trials, substantive changes occurred. Agency watchdogs say this should have called into question the bulk of clinical evidence supporting flibanserin as a safe and effective libido enhancer for women.
And yet, the agency approved the drug for just that use in 2015.
“Yeah, [the FDA] bent over backwards here,” said Erick Turner, a psychiatrist at Oregon Health & Science University who used to review new drugs for the FDA . “And this isn’t the first time I’ve seen this, where they give it to them on close calls.”
In response to queries from Undark, FDA spokesperson Sandy Walsh emphasized in an email that the agency’s priority remains consumer protection first: “Any drug must meet the FDA’s standards for safety and effectiveness,” she wrote, adding that those standards are always modernized and strengthened. “The FDA makes decisions based on consistent, rigorous scientific principles,” Walsh added.
Turner concedes that protocol changes aren’t inherently bad. They can be done for legitimate reasons, and study reviewers are trained to recognize deceptive protocol changes, which must be formally submitted for review. But some changes can invalidate a study just as surely as changing the goalposts in the middle of a game. “The relative timing [of protocol changes] is really important,” he said.
In fact, he and other FDA watchdogs say, the approval of flibanserin is just one of many examples of an agency that can sometimes seem more dedicated to serving the pharmaceutical industry’s interests than those of American consumers. And while the flibanserin case raises methodology questions, other examples — which range from antibiotics with severe side effects to the notorious case of the diabetes drug Avandia — speak to approval of products that are not just ineffective but sometimes dangerous, putting the very citizens the agency is sworn to protect at risk.
Unfortunately, former FDA employees add, fast-tracking new drug products is increasingly part of the agency’s philosophy. “They truly believe that the more you approve and the more you are not-so-diligent about safety issues, the more [a] company will profit,” said Rosemary Johann-Liang Powers, a former FDA medical officer who currently works as a pediatrician in Derwood, Maryland. “And company profiting means they’ll be creating more jobs, they’ll be creating more drugs to get approved, and make available more alternatives for the American people.”
It’s past time, she and other critics say, for the agency to rethink that approach.
“How do we know whether or not they’ve looked at the data?” wondered Turner. “Well, there’s a trust thing there. Sometimes you do wonder.”
In two of the studies submitted for the drug’s approval as a libido booster, the experiments were initially listed as “non-randomized,” meaning that patients were assigned to one of two or more groups that were being compared. By the time the studies were finished, both research teams had abandoned that approach.
Critics say such changes can mean that studies are vulnerable to bias, explicit or implicit, from the researchers conducting them. Turner notes that a suspicious reviewer might wonder if the researchers peeked at the data from their trials, and then switched the methodology around when they didn’t like the results. As it stands, the FDA does not look closely at these changes, Turner said, preferring to trust the scientist.
According to Simon, at the start of the process, pharmaceutical company representatives will typically meet with FDA officials to discuss what they need to demonstrate and how to design experiments in order to get a drug approved. Simon, who worked on the flibanserin case, often works with pharmaceutical companies seeking to get drugs for women’s health on the market. In this case, he started by working on one of the flibanserin clinical trials mounted by Boehringer Ingelheim hoping to extend the drug’s approval from pre-menopausal to post-menopausal women. He then expressed his desire to keep working with manufacturers to help get the compound approved by the FDA even after Boehringer Ingelheim divested from flibanserin.
The trials, he acknowledges, had reporting challenges from the beginning. Participants had to record their levels of sexual desire daily in a small eDiary. Researchers had expressed concern that that such subjective self-reporting is a poor way to measure whether or not flibanserin was increasing sexual desire, but Simon emphasized that the FDA favors enumerated results.
“The FDA wants to count things,” he said, “and the diary is a good way to count. The problem with that is if I ask you whether you’re horny once a day every day and you have to answer, you get a lot of noise.”
The information, flawed or not, helps inform the FDA’s decisions — essentially a regulatory balancing act — on approval of a pharmaceutical product. On one side, the agency weighs the potential benefits of the new drug: the positive effect it has upon patients who are suffering from medical conditions that the drug might help. On the other are the drug’s risks: its side effects and its potential to also do harm. Every drug carries such risks. Only when the benefits outweigh the risks is the agency supposed to allow the new drug to come to market. To some extent, this also reflects a balancing of the protection of consumers against the profitability of the drug industry.
The FDA’s own published analysis notes that the flibanserin process involved navigating between scientific uncertainty and industry pressure. In a perspective piece published in the New England Journal of Medicine in 2016, FDA scientists noted that the agency rejected the drug twice and approved it the third time despite its marginal efficacy — calculated at about 10 percent compared to a placebo effect — because there was no other pharmaceutical stimulant of female sexual desire on the market. The FDA’s approval, also explained in a press release, was announced with something of a lack of enthusiasm. As the press release noted, the drug was being approved only with a “risk evaluation and mitigation strategy” in place, and with a warning label on the box, noting a number of potentially serious side effects.
Such cautions derive from earlier cases in which the agency, the industry, and many thousands of American consumers, paid a heavy price for regulatory lapses that allowed an unsafe drug to enter the market. One of the oft-cited examples of flaws in the process was the 1999 FDA approval of the painkiller, Vioxx or rofecoxib. The pharmaceutical company Merck withdrew that drug from the market in 2004, after it was linked to tens of thousands of cardiovascular deaths. Merck also eventually agreed to pay almost $5 billion to settle initial lawsuits related to those deaths.
Another frequently cited example is that of Avandia, or rosiglitazone, also approved in 1999 for controlling blood sugar in type-2 diabetics. Soon after approval, researchers had discovered that the drug seemed to increase the risk of heart attacks and deaths. However, by hiding scientific information from the FDA, the drug maker, GlaxoSmithKline (GSK), managed to keep the drug on the market and make billions of dollars in Avandia sales.
As later investigations would show, GSK began a clinical trial — Study 175 — in hopes of showing that Avandia was safer than a leading competitor, Actos. To the company’s dismay, the study instead indicated that Avandia was altering patients’ blood chemistry in a way that was much riskier for heart health than Actos was. GSK’s response was to cover up the results of the study rather than publish them and submit them to the FDA, a fact revealed during legal investigations. One email from a GSK executive from October 2000 underlined the point: “Per Sr. Mgmt request, these data should not see the light of day to any outside of GSK.”
As the United States Senate Committee on Finance later reported, “GSK executives intimidated independent physicians, focused on strategies to minimize findings that Avandia may increase cardiovascular risk, and sought ways to downplay findings that the rival drug Actos (pioglitazone) might reduce cardiovascular risk.”
Dr. Steven Nissen, chairman of cardiovascular medicine at Cleveland Clinic and an FDA committee member, can recall in detail the scientific distortion of evidence surrounding Avandia. He was a key player in the case and published research indicating that Avandia was linked with an increased risk of myocardial infarction. His research paper, published in the New England Journal of Medicine, informed the FDA of the drug’s safety issues.
“The politics of this were just unbearably difficult for me because I always thought when I did this whole Avandia thing that it would be about the science, but it’s not always about the science,” Nissen recalled. “It’s sometimes about who’s winning, who’s losing.”
Nissen acknowledged that some of this reflects an often-overwhelmed agency, and that the FDA is not always able to catch such problems if a drug company works to hide negative information from regulators. In this case, he said, the agency would not have realized the extent of the deception “if there hadn’t been a very active investigation of the whole Avandia mess.”
Nevertheless, he also believes that the FDA was aware of the issues with Avandia. “What actually happened here is when the company began to strongly suspect and develop evidence that their drug was harmful, they actually shared the data with the FDA and amazingly the FDA … never publicly disclosed that the company itself had shared with them data suggesting a really serious problem with Avandia.”
This suspicion is reinforced by Powers, the former FDA officer who was involved in reviewing the reports on Avandia. At the time, she was overseeing post-marketing safety analysis at the agency’s Office of Surveillance and Epidemiology. Noting that the heart issue was not even mentioned on containers for the drug, she then pushed for a boxed label warning. “The heart failure issue [revealed in post-marketing analysis],” she said, “was something that was known way ahead of time — even known prior to drug approval.”
Nissen’s research paper made the serious problems with the drug’s approval public —which, he noted, was “intensely embarrassing for the FDA.” The agency went on to conduct its own review, which again found serious health threats related to Avandia and made the drug not available for prescription.
In the end, GSK was fined $3 billion due to the safety issues with Avandia, and a black box warning detailing the concerns was added to the drug’s label in 2007. In analyzing the whole debacle, Nissen concluded that the problems with the drug review and post-marketing review processes were partly about money and partly about hubris. “Hubris was the problem I didn’t fully anticipate — that the FDA would react in such a negative way to something that was implicitly critical of them.”
Meanwhile, in November 2016, experts warned that a new pneumonia drug being reviewed at the time by the FDA, solithromycin, may have severe liver toxicity issues, similar to an old one that shares a related chemical structure. The old drug, telithromycin, marketed as Ketek, was the focus of another series of FDA drug safety scandals in the mid- 2000s. When the drug company Sanofi handed in a large safety study for Ketek, the majority of FDA committee members voted for its approval. But in fact, the study was partly pure fiction and one of its researchers, a physician who fabricated patient enrollment in a small town in Alabama, went on to serve a 57-month sentence in the federal prison.
It also turned out that FDA managers had known of problems with the safety research (known technically as Study 3014) and deliberately avoided mentioning its integrity issue when presenting it to the advisory committee.
In the case of Ketek, David B. Ross, an expert in infectious disease and antibiotics who worked as an FDA drug reviewer from 1996 to 2006, and who is now an associate clinical professor of medicine at George Washington University, said that the drug’s risks clearly exceeded any potential benefits. It was marketed to treat mild infections like bronchitis, bacterial sinusitis, and pneumonia, but turned out to have severe side effects of liver toxicity.
Ross says that while he was at the FDA, he was pressured to alter his negative assessment of Ketek by a division director from the agency. “She called me and asked me to change my review,” Ross recalls. “Her exact quotes were ‘we need more wriggle room in dealing with the company.’” Ross said the director could have written a report to disagree with his decision of not approving the drug. Instead, he felt, she wanted him to change his analysis by ignoring the evidence. It made him feel that this division of the agency “did not have respect for science,” and in 2007, he reported the Ketek case in a perspective article in The New England Journal of Medicine.
That same year, Powers quit the agency, in part due to frustration. “The way data is presented is [not] really the total and unbiased and comprehensive presentation of the information available to date. Even when we get ready for the advisory committee, we are told, ‘don’t present that [data]. That may not be good for the company.’”
In the case of flibanserin, like with Vioxx or Avandia, there are some indications that the FDA and pharmaceutical companies downplayed or ignored safety issues with the drug, which did indeed cause some troubling health problems during the clinical trials.
A close look at the data weighed by the FDA shows that a far greater number of participants suffered from adverse effects while taking flibanserin than the pharmaceutical companies’ reports suggest. One in five participants experienced severe low blood pressure while taking the clinically-recommended 100 milligrams of flibanserin. Another 18 percent experienced the most symptoms such as sleepiness and fatigue after starting at a lower dosage, then working up to 100 milligrams.
One Phase III clinical trial conducted on flibanserin found that more than half of all participants experienced adverse effects, which included dizziness, nausea, headache, and fatigue. To break that down further, the FDA reports that of the 1,791 participants who took 100 milligram dosages, 1,098 had some sort of adverse effect; 121 of which were deemed severe and another 124 of which were so severe that the participant left the study early. These numbers would seem to stand in clear opposition to the “fewer than one percent” adverse effects summary that the FDA presented in another report.
Walsh, the FDA spokeswoman, would not comment directly on the flibanserin approval process, though she argued that many of the practices being cited by critics merely reflect pragmatic decision making. “[The] FDA can use flexible clinical trial requirements and expedited approved authorities,” particularly when a new drug may help with otherwise unmet medical needs, she said. But that is always done, she said, while “prioritizing safety over speed.”
But based on her experience working at the FDA, Powers suggested that varying experience — and ambition — among agency employees also plays a role. “Many of the reviewers — especially medical officers and office directors — do not practice medicine and are very far removed,” Powers explained. “Their goal is, when they leave FDA, most of them would like to find a job at pharma. They don’t want to make their future boss upset.”
As it stands, 46 drugs were approved in 2017. In 2016, there were only 22 drugs approved, and another 45 in 2015, one of which was flibanserin. More than 60 percent of the drugs approved in 2017 have gone through some form of expedited review. Nearly 80 percent of the 46 news drugs were approved in the United States before any other country.
In such actions, Ross identifies a long-time, ever closer relationship between regulators and the companies they oversee. “You have a regulatory agency that’s more concerned with pleasing the industry that it’s regulating,” he said, noting that other examples can be found dating back to the mid-20 century and earlier.
“None of this,” he said, “is new.”
This article was produced by students in the Science, Health & Environmental Reporting Program at the NYU school of journalism.