Astral Codex Ten - Failure To Replicate Anti-Vaccine Poll
I. Steve Kirsch is an inventor and businessman most famous for developing the optical mouse. More recently, he’s become an anti-COVID-vaccine activist. He has many different arguments on his Substack, of which one especially caught my eye: He got Pollfish, a reputable pollster, to ask questions about people’s COVID experiences, including whether they thought any family members had died from COVID or from COVID vaccines. Results here:
All other statistics were normal and confirmed that this was a fair sample of the population. In particular, about 75% were vaccinated (suggesting that they weren’t just polling hardcore anti-vaxxers). Since then, Kirsch has collected several other polls - some by him, some by others - saying the same. For example, here’s Rasmussen (another reputable polling company) from last January: I’ve truncated the table to keep it readable, but I kept the breakdown by party - these aren’t just anti-vax Republicans lying to support their party narrative. Even 19% of Democrats say they know someone killed by the vaccine! I know people here like to argue about whether to debate people or deplatform them, but surely someone else finds this interesting. Here’s 24% of the US - so probably a little short of 100 million people - saying they’ve seen something which consensus science says shouldn’t be possible? Aren’t you at least a little curious what’s going on? Also, isn’t Steve Kirsch being a little too smug about this? So I asked about this on the 2022 and 2024 ACX surveys. Both gave similar results, but I’m going to focus on the 2024 survey, since I did the most followup on it. Here “family” was defined on the question page as including “brother, sister, mother, father, child, aunt, uncle, grandparent, grandchild, niece, or nephew”. This is broader than Pollfish’s “member of your household” but narrower than Rasmussen’s “person you know”. Kirsch and I got similar results for knowing someone who died of COVID - 6.5% vs. 7.5%. But we got very different results for knowing someone who died from the vaccine: Kirsch’s 8.5% vs. my 0.6%. Why? As people love to point out, my survey is a nonrepresentative sample. But as I point out, it’s important to keep track of when that should vs. shouldn’t matter. No matter how weird my readers are, they’re not biologically invincible - they should have side effects at similar rates to anyone else. One possibility is that my readers are very pro-vaccine compared to the general population, so they interpret ambiguous cases in a more pro-vaccine way. I didn’t have a question about vaccine-related views, but it’s no secret that vaccine opponents are more often right-wing, so I looked at questions about politics. Conservatives in general were only slightly more likely (1%) to report vaccine deaths compared to liberals (0.4%). But I had a question where people ranked their support for Donald Trump from. Trump supporters had much higher vaccine injury rates (7.5%) than moderates (1.3%) or opponents (0.3%). I couldn’t find much of an effect by gender, education level, or any of the other traditional demographic categories. This doesn’t quite explain the difference between my survey and the others, since my moderates had 1.3% side effect rate, and the Rasmussen moderates had 22%. But it does suggest that there’s room for political beliefs to alter perception of relatives’ vaccine deaths. II. All of this would be much clearer if we could get in there and ask the people who said their relatives died from vaccines what they meant. Most ACX Survey respondents gave me permission to email them. So I emailed the people who answered “yes” to that question and asked for their story. Some details:
Of the nine people who answered my email, three said they’d read the question wrong and wanted to retract their answer, leaving six people who gave me real stories. I’m slightly obfuscating some of these to protect their privacy:
Four out of six cases are ≥ eighty years old. Since I only heard back from about a quarter of the people who reported deaths, we can speculate that the whole sample had sixteen people in this category. Suppose each of the 6,000 people who took my survey had one relative in this age group (maybe a grandparent). And suppose that each relative got two COVID vaccines (an original and a booster). That means there are 12,000 vaccinations of 80+ year-old relatives in my survey population. The average 80 year old has a 5% chance of dying per year; the average 90 year old a 20% chance. Let’s average it out and say 10% for the 80+ population. That means a 1/3650 chance per day, a 1/500 chance per week, etc. So by coincidence, we would expect about 2 people in my sample to have stories of an older relative dying within one day of vaccination, and about 10 people to have stories of a relative dying within one week. Among people talking about vaccine-related deaths, they seem to range from “same day” to “within six weeks”. So I think this mostly fits the null hypothesis. I don’t think the null hypothesis is quite right here, for two reasons. First, as Respondent 1 notes, some people receive the COVID vaccine in the hospital, when they go there for other reasons, which seems like a time of elevated death risk. Second, Respondent 2 notes that some people are so frail that even normal vaccine side effects of the type that everyone agrees exist might kill them. On the other hand, there might be a tendency for people to wait until they’re healthy to get the vaccine, which would counterbalance these effects. Overall I think the null hypothesis is an okay estimate here though. What about the 37 year old? Yes, he was very obese, but this is still an unusual age to die. Suppose the average respondent has two relatives in this category. Then by a similar calculation, we should expect about 24,000 vaccinations in the population. The average 37 year old has a 1/100,000 chance of dying on any given day. So there’s about a 1/4 chance that we would see an event this extreme in this population (or 1/2 if we interpret “the day after” to mean “not the same day”). But also, since only 1/4 of people answered my email, we should be concerned that there are three other events like this in this sample. This makes it an unlikely, but not extremely unlikely, finding. III. I interpret the results of my survey to be consistent with a null hypothesis of “the vaccines don’t increase deaths”, plus or minus some very small effect of “they can kill the extremely frail” or “they can cause a heart attack in susceptible young people 1/10,000th of the time”. That still leaves the question of why my results are so different from those of Kirsch, Pollfish, and Rasmussen. Maybe there’s some very big population of people who got the vaccine, then died eventually (where “eventually” could be anything from the same day to years later) of some condition (where “some condition” ranges from things plausibly connected to vaccines like allergic reactions, to things not plausibly connected to vaccines like lightning strikes). Depending on people’s previous assumptions about the risks of the vaccine, they’ll either report these as vaccine-related deaths, or think of them as unrelated coincidences. This would explain why my data found that Trump supporters were 20x more likely than Trump opponents to know a vaccine-related death. But this can’t be the whole explanation: ACX readers are only a little further left than the general population, and even the most pro-Trump ACXers reported lower death rates than the median Kirsch poll respondent. My guess is that this is a blog about statistics and reasoning, so people here are very cautious about the post hoc ergo propter hoc fallacy. I think the best conclusion from this result is just to stop caring about these kinds of polls. Any poll whose outcome can change by more than an order of magnitude based on the respondents’ politics or statistical knowledge isn’t a valid guide to the frequency of real-world events. This should have been our leading hypothesis all along, but the results were weird enough to be worth checking. Now that we’ve checked, we can forget about this methodology and focus on the result of peer-reviewed studies, like we should have done all along. As always, you can try to replicate my work using the publicly available ACX Survey Results. If you get slightly different answers than I did, it’s because I’m using the full dataset which includes a few people who didn’t want their answers publicly released. If you get very different answers than I did, it’s because I made a mistake, and you should tell me. You're currently a free subscriber to Astral Codex Ten. For the full experience, upgrade your subscription. |
Older messages
Nobody Can Make You Feel Genetically Inferior Without Your Consent
Thursday, June 13, 2024
... ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Open Thread 333
Monday, June 10, 2024
... ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Berkeley Meetup This Wednesday
Tuesday, June 4, 2024
... ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Open Thread 332
Monday, June 3, 2024
... ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Open Thread 331
Monday, June 3, 2024
... ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
You Might Also Like
🏆 The One 2024 Performance To Rule Them All
Monday, December 23, 2024
Plus: Inverse spoke to the cast of 'The Brutalist' about the film's unlikely beginnings, and that ambiguous ending. Inverse Daily Charlie Vickers' transformative performance as Sauron
☕ Ghosts of elections past
Monday, December 23, 2024
Did the 'AI election' come to pass? December 23, 2024 View Online | Sign Up Tech Brew presented by Nasdaq It's Monday. Hello and welcome to Holiday Week, the hazy days between Christmas and
The McKinsey Opioid Consultants Who Got Away
Monday, December 23, 2024
The Biden administration just delivered a sweetheart deal to its friends at McKinsey, letting the consulting giant off the hook for helping to “turbocharge” disastrous OxyContin sales. Investigators
The real story behind the holiday’s most annoying carol
Monday, December 23, 2024
Plus: What the government shutdown drama reveals about the incoming administration, and more. December 23, 2024 View in browser Editor's note: We'll be taking a few days off for the holidays,
The real story behind the holidays' most annoying carol
Monday, December 23, 2024
Plus: What the government shutdown drama reveals about the incoming administration, and more. December 23, 2024 View in browser Editor's note: We'll be taking a few days off for the holidays,
Friendly Fire Over Red Sea, Manchin's Warning, and Holiday Party Tips
Monday, December 23, 2024
US forces shot down one of its own planes over the Red Sea on Sunday, forcing the two Navy pilots on board to eject, with one sustaining minor injuries. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
☕ The list of lists
Monday, December 23, 2024
A roundup of the best stuff that got made in 2024... December 23, 2024 View Online | Sign Up | Shop Morning Brew Presented By Pacaso Good morning, and Happy Festivus! Before we get the pole out of the
Numlock News: December 23, 2024 • Moxie, Malls, Grand Marnier
Monday, December 23, 2024
By Walt Hickey ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Open Thread 361
Monday, December 23, 2024
... ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
The Automattic vs WP Engine WordPress wars are getting really annoying [Mon Dec 23 2024]
Monday, December 23, 2024
Hi The Register Subscriber | Log in The Register Daily Headlines 23 December 2024 wordpress The Automattic vs WP Engine WordPress wars are getting really annoying Forks at dawn.... but it's not