In 1978, an unusual documentary appeared on American television. Narrated by Peter Falk, “Scared Straight!” chronicled a group of juvenile delinquents who spent a day with hardened convicts in Rahway State Prison (now East Jersey State Prison) in New Jersey. The intervention was intended to scare the teens away from a life of crime by exposing them to the brutal realities of life behind bars, and it seemed successful: the filmmakers found that most of the teenaged participants did not reoffend. “Scared Straight!” won the Oscar for Best Documentary Feature, engendered a wave of Scared Straight programs for young offenders in many states, and inspired the A&E reality series “Beyond Scared Straight,’’ which ran from 2011 to 2015.
BOOK REVIEW — “Randomistas: How Radical Researchers Are Changing Our World,” by Andrew Leigh (Yale University Press, 288 pages).
The only problem was that it didn’t really work. Shortly after “Scared Straight!” aired, a Rutgers criminologist conducted a randomized trial of teens who went through the Rahway program (not just the handful in the documentary) and found they were actually more likely to commit crimes than their peers who didn’t. The Rutgers study received little attention, nor did later studies suggesting that Scared Straighters were 1.68 times more likely to reoffend. Many Scared Straight programs continue today.
No one is certain why these programs seem to encourage recidivism, but it’s pretty clear why American society embraced the film. It told a story that appealed to common sense: If “at risk” teens could get an up-close view of what awaited them down the road, surely they would change course. But, as Andrew Leigh demonstrates in his new book, “Randomistas: How Radical Researchers Are Changing Our World,” we’ve always been suckers for common sense explanations.
Leigh, a former economist at Australian National University, fills his account with tantalizing examples that reinforce the Scared Straight lesson: When we go with our gut, we’re often wrong. Making teenage girls care for a demanding baby doll does not diminish, but according to one study actually doubles their risk of teen pregnancy. (Those baby dolls are cute.) Advancing microcredit in the developing world does not significantly raise household income. (It just puts people in debt.) Annual physicals do nothing to prevent illness.
All those findings were undermined by randomized controlled trials, in which a group is randomly divided into two or more subgroups that receive different treatments or interventions. Randomized trials are considered the gold standard of clinical research for their ability to minimize selection bias, and today they are de rigueur in the fields of medicine and, increasingly, economics and sociology. But as Leigh makes clear with his subtitle (chosen, the author concedes, after a randomized online trial that evaluated the clickability, of various options), that wasn’t always the case.
The earliest randomized trial Leigh cites is that of James Lind (shown in the image above), a Scottish ship’s surgeon who in 1747 randomly assigned his sailors six different scurvy tonics: sulfuric acid (the standard Navy treatment of the day), cider, seawater, vinegar, oranges and lemons, and a gut-clenching muddle of nutmeg, garlic, horseradish, mustard, and other spices. Lind’s results, published in his 456-page book “A Treatise of the Scurvy,” were unequivocal: the sailors who got the citrus were back on their feet in a few days.
This should have rocked the empire. At the time, scurvy killed scores of sailors. In the Seven Years’ War (1756–63), just 1,512 of Britain’s 185,899 sailors died in combat, while scurvy took 133,708. And yet neither the medical world nor the Navy acted on Lind’s finding. It took another 50 years before the Navy made lemon juice a part of standard rations.
This is an ongoing theme in “Randomistas”: Even when a randomized trial undercuts conventional wisdom, we often stick with our beliefs. After-school programs haven’t been shown to improve academic performance, and may actually increase students’ odds of being suspended (bad crowd?), yet we still spend more than $1 billion per year on them. An Australian study published in 2015 of severe head trauma cases compared victims who received paramedics as first responders with those treated by a physician flown to them by helicopter. The study found no difference in outcomes, but the government still chose the much more expensive practice of flying out a physician whenever possible.
Part of the problem is that randomized trials don’t tell us why something works, or doesn’t. Like machine learning, the trials have a bit of a black box problem: They reveal a connection, but they don’t tell us anything about how the world works.
For that and other reasons, randomized trials have drawn a fair amount of criticism. The term “randomistas” was actually coined by the Nobel laureate economist Angus Deaton to disparage those who put too much stock in randomized trials of development programs. Randomized trials are often seen as bias-free, but they always have their own assumptions baked in, and their results are only an average of the group studied. Could it be that microcredit does work for certain people in certain situations, given appropriate times and terms?
Of greater concern is the issue of replicability. In recent years, it has become clear that many, if not most, published research findings cannot be duplicated. Many of these were based on randomized trials. The problem is not so much in the trials themselves, but our tendency to treat their results as definitive. Leigh suggests several ways to improve the rigor of trials — run your trial in two or more locations (preferably in different countries) among very different groups; pre-register all studies so the ones that fail don’t disappear.
If this can help restore our faith in randomized trials, “Randomistas” will have provided a valuable service, for Leigh makes a convincing case that 21st-century humans continue to cling to disproven beliefs as surely as the 18th-century British Navy. He shows this most effectively in the realm of health care, where randomized trials have suggested zero benefit from multivitamins and omega-3 supplements, and a harmful impact from hormone therapy and breast cancer screening. Yet these practices aren’t going away anytime soon.
And then there’s surgery, the area of medicine most immune to randomized trials, which would require that a control group be anesthetized, cut open, and sewn up without having anything fixed inside. It sounds absurd, but many such trials have been performed, and the results are startling. The most common orthopedic procedure in the United States is to repair a torn meniscus — the cartilaginous pad in the knee — yet a 2013 study found it no more effective than “sham” surgery. Shockingly, that’s no outlier: In a review of 53 sham surgery trials, real surgery bested the placebo only half the time.
Gems like that are the book’s strength. Unfortunately, too much of “Randomistas” reads as little more than a grab-bag of examples, never building to any insightful synthesis. It wants to be another “Freakonomics,” but instead of revealing counterintuitive connections that help us see the world in new ways, it simply makes its point — that controlled trials can help us overcome bias — over and over.
Still, many of the examples are remarkable enough to make you a terror on the cocktail party circuit for months afterward. And if you can prevent even a single mariner from ingesting sulfuric acid on his next sea voyage, it will be time well spent.
Rowan Jacobsen is a journalist and author, and a former Knight Science Journalism Fellow at MIT. His books include “American Terroir,” “The Living Shore” and “The Essential Oyster.”
Comments are automatically closed one year after article publication. Archived comments are below.
Read the old book: Microbe Hunters by Paul DeKruff. He tells of scientific and medical discoveries which were ignored and those who made them ridiculed. Lister, for one, who encouraged carbolic acid as a germ killer. I read it decades ago when I was in high school and it made me a skeptic about people’s ability to even look at evidence which would change their opinions.