Attempts To Put Statistics In Context, Put Into Context
Sometimes people do a study and find that a particular correlation is r = 0.2, or a particular effect size is d = 1.1. Then an article tries to “put this in context”. “The study found r = 0.2, which for context is about the same as the degree to which the number of spots on a dog affects its friskiness.” But there are many statistics that are much higher than you would intuitively think, and many other statistics that are much lower than you would intuitively think. A dishonest person can use one of these for “context”, and then you will incorrectly think the effect is very high or very low. In last week’s post on antidepressants, I wrote:
But we can do worse. Studies find that IQ correlates with grades at about 0.54. Here are two ways to put that in context:
The first way makes it sound like IQ doesn’t matter that much; the second way makes it sound like it matters a lot. Or suppose that you’re debating whether there’s such a thing as “general” intelligence, eg whether students who are good at reading are also good at math. People have studied this; here are two ways to describe the result:
The first effect might sound kind of trivial, but it is r = 0.86. And the second effect might sound immense, but it is only r = 0.64. The real correlation between standardized reading and math tests is in between, r = 0.72. The examples above might be sort of cheating, because they’re comparing college majors (which are averaged-out aggregates of people) to countries (which are just individual countries). But that’s my point. It’s easy to cheat! Obviously someone wanting to exaggerate or downplay the generality of intelligence could choose which of these two ways they wanted to “put it into context”. I don’t have a solution to this except for constant vigilance and lots of examples. So here are a lot of examples. I thought I was the first to do this, but partway through I found some prior art. None completely satisfied me, but I’ve stolen a little from all of them. Credit here: Meyer et al, Hattie on education, Reason Without Restraint, Leucht et al. Some effect sizes and correlations are naturally misleading, or depend a lot on context. I’ve tried as hard as I can to avoid these and make all my examples clear, but they will necessarily require some charity. Effect Size:
Correlation:
A statistician who read a draft of this post suggested throwing out these general statistical effect measures in favor of specific ones; not only does nobody know what they mean, but nobody cares what the “standardized effect size” of a treatment for depression is - they care how much less depressed they’re going to get. I sort of agree, but I can see some use for these. We can’t express how much less depressed someone gets in comprehensible units - it will either be stuff like this or “5 points on the HAM-D”, which is even more arcane and harder for laymen to interpret. And when you get to questions like “how big is the gender difference in empathy?”, I can’t think of another equally clear way to express it. Still, it’s hard to understand and easy to mislead with, so watch out. You're currently a free subscriber to Astral Codex Ten. For the full experience, upgrade your subscription. |
Key phrases
Older messages
Your Book Review: Man’s Search for Meaning
Monday, June 12, 2023
Finalist #4 in the Book Review Contest
Highlights From The Comments On The Academic Job Market
Tuesday, June 6, 2023
...
Open Thread 279
Monday, June 5, 2023
...
Your Book Review: Why Machines Will Never Rule the World
Saturday, June 3, 2023
Finalist #3 in the Book Review Contest
Links For May 2023
Thursday, June 1, 2023
...
You Might Also Like
Numlock News: May 13, 2024 • Apes, Arts, Automation
Monday, May 13, 2024
By Walt Hickey Welcome back! It's the Numlock sixth anniversary this month! Get a steeply discounted subscription today: Apes Kingdom of the Planet of the Apes, the 10th installment in a 56-year-
The California Exodus, a Senator Says He Was Wrong, and a 106-Year-Old Jumps Out a Plane
Monday, May 13, 2024
Hundreds of California law enforcement officials have grown tired of the state's “soft-on-crime” policies and relocated to Texas, according to one media outlet. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
☕ 100% tariffs
Monday, May 13, 2024
The northern lights dazzle around the world... May 13, 2024 View Online | Sign Up | Shop Morning Brew PRESENTED BY Slack Good morning. Heads up: Later today, OpenAI will announce “new stuff” that “
Open Thread 329
Monday, May 13, 2024
... ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
GhostStripe attack haunts self-driving cars by making them ignore road signs [Mon May 13 2024]
Monday, May 13, 2024
Hi The Register Subscriber | Log in The Register {* Daily Headlines *} 13 May 2024 stop GhostStripe attack haunts self-driving cars by making them ignore road signs Cameras tested are specced for
Welcome to The Flyover
Monday, May 13, 2024
Thanks for joining The Flyover! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
October 7 Survivors Sue Campus Protesters, Say Students Are “Hamas’s Propaganda Division”
Sunday, May 12, 2024
Four lawsuits alleging Hamas ties against Students for Justice in Palestine, the AP, UNRWA, and a cryptocurrency exchange share many of the same plaintiffs. Most Read Criticizing Israel? Nonprofit
Monday Briefing: Russia pushes into northern Ukraine
Sunday, May 12, 2024
Also, fighting is raging in northern Gaza View in browser|nytimes.com Ad Morning Briefing: Asia Pacific Edition May 13, 2024 Author Headshot By Amelia Nierenberg Good morning. We're covering
Guest Newsletter: Five Books
Sunday, May 12, 2024
Five Books features in-depth author interviews recommending five books on a theme. Guest Newsletter: Five Books By Sylvia Bishop • 12 May 2024 View in browser View in browser Five Books features in-
GeekWire's Most-Read Stories of the Week
Sunday, May 12, 2024
Catch up on the top tech stories from this past week. Here are the headlines that people have been reading on GeekWire. ADVERTISEMENT GeekWire SPONSOR MESSAGE: Join the Nordic Innovation Summit, May 13