I: Pretending
Let’s pretend the hype cycle of “party drugs are miracle mental health drugs!” is already over. This time wasn’t different. The rates of mental health problems are either staying unacceptably high or climbing even higher. The well-intentioned1 scientists, once hopeful patients, opportunistic grifters, and bored VCs have all moved on to the next big thing.
How could we have gotten here?
After all, there was just a New York Times writeup of a gold-standard, placebo-controlled trial showing MDMA improved PTSD! The headline even says “A Psychedelic Drug Passes Big Test for PTSD treatment.” There have been a lot of other promising headlines too, like “Could the Embrace of Psychedelics Lead to a Mental-Health Revolution?”, “Mind Menders: The Future of Psychedelics for Mental Illness”, and “How ecstasy and psilocybin are shaking up psychiatry.” That last headline is from Nature.
Humans are bad at predicting the future. But if the party drug hype cycle ends, a lot of its downfall could be predicted by one of the first skills we develop.
II: Counting
There’s a great statistics textbook I’m slogging my way through right now. A lot of its insights have come from translating scientific jargon into actionable English. At many points throughout, the author reminds us statistics is glorified counting.2
Let’s do some counting with the latest, greatest party drug trial. From the NYT article, “67 percent of participants in the MDMA group no longer qualified for a diagnosis of PTSD, compared with 32 percent in the placebo group.” That sounds super impressive! But percentages aren’t quite counting yet.
There were 90 people in the trial, so let’s assume an equal number of people in both groups and that literally no one left the study.3 That means ~30 out of 45 people who took MDMA got better, while ~15 out of 45 people who took an inactive placebo got better.
I’m over the moon for the ~45 folks who got better during this study, including ~30 of them who did so in the group who took MDMA. And if you’re also asking “Wait, only a high school classroom’s worth of people got better with MDMA? Is that enough to decide a drug works?” your instincts are right.
III: Powering
Part of the alchemy that turns counting into helpful statistics is having enough stuff to count. Every single person’s life is valuable beyond measure. In a randomized trial of a treatment each person counts exactly once.4 We can do a power analysis — a formula that tells us how many people we need to trust our counting process — to figure out if the party drug trial had enough stuff to count.
When we run this formula and assume party drugs have effects similar to psychotherapy and other psychotropic medications5 we find this trial had nowhere near enough stuff to count. The study would need to be ~100 times larger for us to take the counts seriously.
If, however, we assume MDMA is twice as powerful as any other high-quality intervention we’ve ever seen, there’s more than enough stuff to count. We’d only have count up the responses of 78 folks to know reliably how well people responded to MDMA.
IV: Pretending Again
Now, we’re back to pretending the party drug hype cycle is over. It’s obvious now we shouldn’t have expected these drugs to be twice as powerful as anything we’ve tested before. Sure, the smaller studies showed much larger effects, as they always do. But in this world we conducted much larger trials next, and the apparent benefits shrank or disappeared. Or we never conducted larger trials, but it’s been years to decades with no obvious improvement in overall mental health.
We’re looking back with disgust at the enrollment of patients into studies without telling them, nodding knowingly about the many documented, non-counting related flaws in the “above board” studies, and already hoping the next hyped treatment will succeed where party drugs have failed. The New York Times has even written an excellent investigative piece tying together everything that went wrong. The people they interview who are still suffering break our hearts.
V: Hoping
I hope the world where the party drug hype cycle ends is only a fantasy. I hope against hope, despite no good evidence, that party drugs are twice as effective (or more) as anything we already have. I’m human, I hope for a lot of things that probably aren’t true.
Or maybe not, who knows↩︎
The author is making a specific, technical point in the context of Bayesian statistics, and I think the general principle is still useful↩︎
That seems unlikely, but let’s give the study the benefit of the doubt↩︎
Unless you have lots of repeated measures from individual people, but that’s just a different, more complicated kind of counting↩︎
Using d = 0.20 as an approximation from low risk of bias, higher sample size trials testing these treatments, see https://www.ncbi.nlm.nih.gov/books/NBK78732/, assuming the placebo response rate was equivalent to what was observed in the trial, and only giving ourselves a 5% chance of a false negative↩︎