The title of this post is inspired by Scott Alexander’s Never Tell Me The Odds (Ratio). The goal of this post is to explain the meanings of (commonly-heard) metrics that indicate the “odds” of something (either directly or indirectly).
Just because these terms are commonly-heard does not mean they are commonly-understood. The odds are that most people don’t understand the numbers related to the odds – and misinterpret how big the odds really are.
Let’s take an example, borrowed from Scott Alexander:
Suppose you run a drug trial. In your control group of 1000 patients, 300 get better on their own. In your experimental group of 1000 patients (where you give them the drug), 600 get better.
The relative risk of recovery from the drug = probability of recovering from the drug in the experimental group ÷ probability of recovering on one’s own in the control group = (600 / 1000) ÷ (300 / 1000) = 60% ÷ 30% = 2.0.
The odds from recovering from the drug in the experimental group = probability of recovering ÷ probability of not recovering = 600 ÷ (1000 – 600) = 3/2. Likewise, the odds from recovering on your own in the control group = 300 ÷ (1000 – 300) = 3/7.
The odds ratio = odds of recovering from the drug ÷ odds of recovering on one’s own = (3/2) ÷ (3/7) = 3.5.
The Cohen’s d effect size takes the difference in the average of two groups (x1 – x2) and divides it by the standard deviation (s):
(Formula screenshots taken from this post on effect size.) Cohen’s d for the example above = (0.6 – 0.3) / 0.474341 = 0.6. I have used this standard deviation calculator and this Cohen’s d calculator. Note that Scott Alexander’s result is a little bit different at 0.7.
To recap, for the example above, we got the following results:
- Relative risk (drug vs. self-recovery) = 2.0
- Odds ratio (drug vs. self-recovery) = 3.5
- Cohen’s d effect size = 0.6
The numbers lie on a wide range from 0.6 to 3.5 – and depends on which one is reported, and in what fashion, it could bias up (or down) the reader’s perception on how effective the drug is (vs. self-recovery). As Scott Alexander puts it:
The moral of the story is that (to me) odds ratios sound bigger than they really are, and effect sizes sound smaller, so you should be really careful comparing two studies that report their results differently.
May the odds be forever in your favor! 😉
Fully Booked Club 