(Lack of) Evidence for pandemic NPIs
A response to Dr. Tom Frieden's article in WSJ + Q&A about our new publication on pandemic RCTs
Dr. Tom Frieden’s new article in the WSJ made me consider how little good evidence we actually have the interventions he praised as lifesaving during the pandemic were effective. He draws a lot of conclusions from confounded observational data & early modeling studies. The reason I bring it up is it bothers me that many people will look back on the pandemic thinking we had good evidence masks, lockdowns, or as he says “closure of indoor activities”, etc “worked.” We didn’t have good evidence and we don’t. And that’s important because if there is not widespread acknowledgement of that, we will continue to make inappropriate decisions based on the same sort of weak data in the future.
But first I have to say, he got this right: “schools were closed when they could have remained open”
Thank you. I have been saying that since Spring of 2020. But whatever it is those kids are doing.. that picture (they call it the “zombie walk” in the caption) should go straight to the Museum of Pandemic Absurdities.
But how did he draw this conclusion for schools & not the other NPIs? Ss for the masks and selective closures decreasing excess mortality, he seems to have entirely forgotten about the country of Sweden, which only limited gatherings to under 50 people and had very brief secondary school closures and very limited mask wearing and one of the lowest excess mortality rates in Europe.
He mentions this modeling study from New York City, which concluded all of the below measures “worked’ to some degree, but they have no comparison within the city to areas that did not close schools, issue stay-at-home orders, advise or mandate masks. They actually could have randomized different areas to different interventions.
And, no that would not have been unethical, especially knowing now better evidence has suggested closing schools and mandating masks did not work. What may have been happening here was we were seeing this was a seasonal respiratory virus or cases may have just been decreasing on their own.
Europe’s cases also decreased that Spring but Sweden’s death rate from covid-19 declined more slowly- the question is why. What was it about their response that led to this? Something with nursing homes? We are left wondering.
And one thing that interests me about a study from Europe he cites as evidence lockdowns worked is the lack of dose response to the interventions. “First”, which I understand is “early”, did not have more impact than later enactment of the below interventions.
And, in the end, looking at pandemic age-standardized excess mortality, Sweden may have not done as well in the first mile of the race but did not lose the marathon when comparing it to other European countries or even its Nordic neighbors. By Paul C. I don’t like to make predictions, but it seems unlikely they’ll lose the ultramarathon either. Health was never just about one disease.
And what about the vaccination data he cites for effectiveness against death from COVID-19?
The data are confounded by better underlying health (at least 3-4x less likely to die from non-covid causes) in the more highly vaccinated in the US and skewed by the fact people with unknown vaccination status are considered “unvaccinated” and are more likely to be uninsured/lower SES etc. So is 10x less likely correct? The answer is- probably not given healthy vaccinee bias. We must be honest, that without any randomized data - or even falsification endpoints on the bivalent boosters, we are currently flying blind in terms of estimating their effectiveness against severe disease and death from COVID-19. All of the data we have are observational and when they are not reported with non-covid death or hospitalization rates by vaccination status need to be assumed to be confounded by healthy vaccinee bias until proven otherwise.
Then he presents evidence he believes shows masks “worked”
But what he cites are many confounded observational studies. He fails to cite any RCTs except Bangladesh, the results of which have been found to be too uncertain due to sampling bias to be considered significant; there are issues beyond sampling bias as well. He also fails to cite the best peer reviewed study we really can get causal inference from - the regression discontinuity study from Spain, which found no effect of mask mandates on infection or transmission. He also does not cite all of the pre-pandemic randomized studies for influenza/influenza like illness that were negative!
But the above quote is what really got me. He says: “randomized controlled trials aren’t usually the best way to evaluate communitywide interventions, particularly during a pandemic.”
Okay- admittedly, that might be the case if you had good evidence of effectiveness- and particularly that the benefits outweigh the harms. But that’s simply not what we have or had for all of the pandemic NPIs. None of them. A very important part of public health is stating what we don’t know - and could have learned- but didn’t. And the mere fact we did randomized studies of NPIs during the pandemic shows we can do them.
—-Which bring me to the topic of a new peer-reviewed article I wrote with Vinay Prasad. It’s entitled “An evidence double standard for pharmacological versus non-pharmacological interventions: Lessons from the COVID-19 pandemic” and poses the (I thought) straight-forward question of why we require randomized trials of pharmacological treatments but, when it came to non-pharmacological interventions during the pandemic (school closures, lockdowns, mask mandates), they were not only recommended but required- often long term- without good evidence they worked.
Where did this assumption come from that COVID-19 NPIs would obviously work so well that their effectiveness would outweigh any harms? From wishful thinking? Is that what led many to accept studies of two hair stylists and phone surveys as good evidence?
When you consider how medical evidence typically works, it is generally assumed putting a pill in your mouth might not have the intended effect and can have side effects, so the efficacy and harms must be established before use is recommended. Why was it not assumed that interventions like school and business closures would also come with harms? Hubris? Unwillingness to acknowledge most things we think will work - and are even very plausible - don’t?
As we said, and Dr. Michael Absoud highlighted, NPIs can have wide societal consequences, which we undoubtedly saw with school closures and lockdowns, and should not be assumed to be of less consequence than side effects from medications.
As we mention in the article, Atle Fretheim and colleagues in Norway called for RCTs of school closures. However, Norway’s school closures were so brief - only 6 weeks; this may be why no randomized studies were undertaken. Because most of Europe reopened school in the spring and summer with observational data failing to note a surge in the community, schools there, for the most part, remained open.
But the US and much of the world outside of Europe had prolonged school closures and, honestly, the return to brief school closures time and again across Europe (though not in Sweden) with the emergence of new variants was never evidence-based. With data from Germany and Brazil showing no correlation between school closures and COVID-19 cases or mortality (the latter studied in Brazil), it has only become clearer with time the harms of school closures vastly outweighed any undetected benefits. And this is an example of what can happen when humans make evidenceless assumptions and force people to comply. This is why we should require robust evidence of effectiveness of NPIs (just as we do for pharmacological interventions) before they are recommended, let alone mandated.
While most astute readers seemed to understand the argument our paper made, some of the comments to my post on Twitter surprised me, but maybe maybe others have similar questions.
Take this comment:
My favorite question is “can cups hold water?” TheyCallMeTarz is describing things we can observe the nature and effectiveness of directly with our senses (in this case mostly just with our eyes). I think essentially everyone would agree we don’t need an RCT when we can just use our eyes.
Also, I greatly prefer water without ice. But that’s sort of beside the point.
It’s the same for parachutes and not jumping in front of a moving bus. You can see immediately these things work. You don’t need a randomized study to tell you . This is simply not true for masks or closures or CO2 monitors because, even if mechanistically they “work”, humans are gonna human (a modification on Zeb’s virus gonna virus, of course).
And we have a lot of observational data that places with mask mandates and school closures did not do any better at curbing COVID-19 spread than those that didn’t. And there was and is significant debate about whether or not they would work (equipoise), so RCTs were indicated if they were to be implemented.
But this gets to another critical part of our article: who needs to generate the evidence to justify recommending or imposing an intervention? Of course it is the people who are recommending or requiring it.
And this is the point Dr. Lam (below) may be missing. It is public health officials who need the data to justify their recommendations. It is not up to the people who are not pushing the recommendations to do the studies.
There is a very long list (infinite, I suppose) of things I don’t recommend and also don’t do RCTs of. I do however commend the researchers who have done RCTs of NPIs. I was surprised when writing our article to see the RCT of CO2 monitors for maintaining CO2 below a certain level in hospital rooms… the RCT was negative; the intervention did not work because the CO2 levels were no different in the rooms with the monitors (and instructions given to keep below a certain level) and those without. The authors state one of the reasons was patients got too cold with the windows open.
But more to Dr. Lam’s point, of course RCTs for mask mandates have been designed and run and were deemed ethical by IRBs. Just look at the two Cochrane reviews of randomized studies for masks. And I was recently in a discussion on Twitter about the opportunity of studying masks in the healthcare setting as hospitals drop mask mandates… and I said “yes, good idea!” This would indeed be a good time and opportunity to do an RCT! So, it’s not like these studies haven’t been or can’t be done. As long as there is equipoise they are ethical and should be considered necessary when existing observational data are insufficient and public health officials wish to recommend the intervention for widespread use.
But here we are, 3 years later and we never generated high quality evidence the NPIs that were recommended (and even forced) worked.
On a different note, congrats to all the senior med students who just matched! Crazy to think I went through the match twice matching into ophthalmology and then PM&R - both great for nerding out. If you are doing residency in PM&R you are in for such a treat because we see such a wide range of diseases and problems and really get to know our patients well. Delightful. Just ran across this photo from 5 years ago during residency at UC Davis lecturing on one of my favorite topics- EMG spontaneous activity
Be happy, my friends. Do what you love and stand up for what you believe is important; sometimes that might mean generating the evidence for or saying we don’t know how well society-wide interventions work- and that harms may outweigh the benefits; there are many ways for us physicians to speak up on behalf of our patients.
Okay, before I end, what did I get wrong? I appreciate being corrected.
Thank you Dr. Hoeg. Is there any chance you can get this published in the WSJ? Though its’ an opinion piece, most people reading it will not take that into consideration and take his views as fact.
Thing is, public health policymakers can ignore the evidence you and many accomplished experts put forward and assert their own flimsy stand-ins for evidence for as long as they want. And stand in their definition of ethics doing it. And have courts uphold their policies. Take the practice of water fluoridation. Same flimsy evidence asserted and upheld in the face of large bodies of evidence produced by many accomplished experts. Upheld for over 75 years, coercive public health actions that remove consent from the public, no opt-outs where judges haven't put a stop to the mandates. They call it an ethical, "good stewardship" model.
The Nuffield Council on Bioethics produced this guide to ethical public health policymaking in 2007. It is considered a gold-standard presentation of bioethics used by public policymakers internationally to help guide their decisions. It's guidance is still relevant and useful today, explaining much of what public health officials have relied upon to determine public policies. Subjects it touches on include obesity, tobacco and alcohol, infectious disease and water fluoridation. That doesn't necessarily make it a good guide.
https://www.nuffieldbioethics.org/wp-content/uploads/2014/07/Public-health-ethical-issues.pdf
I find Chapter 7, Water Fluoridation beginning on page 121 (153) useful in the context of community masking mandates. It all is useful to read to gain full context, and beginning on page 135 (167) is a section on Consent that gets to the principles that inform public policymakers on the ethics of coercive public health actions that remove the consent of the public, no way to opt-out.
Throughout the entire guide and even in the section on Fluoridation it gives voice to individual free choice. That it rarely honors in its final subject guidance. With water fluoridation it shares that even after over 60 years (in 2007, now over 75 years) of fluoridating water there's little compelling evidence that it is effective. It acknowledges that in lab studies some results have been promising, but clear scientific results have never been produced that show it works as claimed in a community setting. It also acknowledges that opponents of fluoridating water have produced lab studies showing harmful effects, including increased cancer, from fluoridated water, but they've been disputed by proponents of fluoridation and so there's no clear scientific results that shows it is unquestionably harmful. Earlier in the guide it makes the case that no ethical public policy should be made so coercive it removes consent unless it is proven to be safe and effective.
With regards to fluoridation it says that because there's no clear, compelling evidence on either side that nation's ought not make it public policy for ethical reasons. *But* it goes on to offer that if local communities wish to do so they may. Then shifting the burden of proof onto opponents of fluoridation that shows clear, unquestionable scientific evidence it is unsafe. Violating the very ethics of the guide it purports to adhere to.
Apply this same reasoning to pandemic mitigation public policy, including mandatory mask mandates, policies that remove individual consent for a proclaimed collective public health concern. Know that water fluoridation has over 75 years of practice now in many communities, with no compelling evidence it is effective at preventing cavities (caries). But based on mere assertions of efficacy local communities fluoridate water. And because opponents aren't able to produce scientific evidence of harm that satisfies the proponents of fluoridation it is deemed ethical public policy at the local community level. With this model as a guide, local communities could continue with mask mandates for 75 years without any compelling evidence, dismissing all evidence of harms presented by opponents. And consider themselves ethical. Even "Good Stewards" of their citizens.
This bioethics guide proclaims its product represents the ideal of "Good Stewardship" that balances the competing interests of individual choice and freedom with the collective's public health and safety required sacrifices. In fact, it gives mere lip service to individual choice and freedom as it minimizes the value of it, while giving broad deference to presumptions of necessary collective sacrifices with little scrutiny. But because it gives any kind of voice to the individual it pats itself on the back for balancing the competing interests. It's mental masturbation for the creators of it, with multiple exclamations of King Solomon-like wisdom.
And it's significant to note that the head of Bioethics at the National Institute of Health is Christine Grady, wife of Anthony Fauci. Mic drop.
https://thenationalpulse.com/2022/05/18/fauci-wife-authors-paper-supporting-vaccine-pressure-campaigns/