The title of this post is inspired by Scott Alexander’s Never Tell Me The Odds (Ratio). The goal of this post is to explain the meanings of (commonly-heard) metrics that indicate the “odds” of something (either directly or indirectly).
Just because these terms are commonly-heard does not mean they are commonly-understood. The odds are that most people don’t understand the numbers related to the odds – and misinterpret how big the odds really are.
Let’s take an example, borrowed from Scott Alexander:
Suppose you run a drug trial. In your control group of 1000 patients, 300 get better on their own. In your experimental group of 1000 patients (where you give them the drug), 600 get better.
The relative risk of recovery from the drug = probability of recovering from the drug in the experimental group ÷ probability of recovering on one’s own in the control group = (600 / 1000) ÷ (300 / 1000) = 60% ÷ 30% = 2.0.
The odds from recovering from the drug in the experimental group = probability of recovering ÷ probability of not recovering = 600 ÷ (1000 – 600) = 3/2. Likewise, the odds from recovering on your own in the control group = 300 ÷ (1000 – 300) = 3/7.
The odds ratio = odds of recovering from the drug ÷ odds of recovering on one’s own = (3/2) ÷ (3/7) = 3.5.
The Cohen’s d effect size takes the difference in the average of two groups (x1 – x2) and divides it by the standard deviation (s):
To recap, for the example above, we got the following results:
Relative risk (drug vs. self-recovery) = 2.0
Odds ratio (drug vs. self-recovery) = 3.5
Cohen’s d effect size = 0.6
The numbers lie on a wide range from 0.6 to 3.5 – and depends on which one is reported, and in what fashion, it could bias up (or down) the reader’s perception on how effective the drug is (vs. self-recovery). As Scott Alexander puts it:
The moral of the story is that (to me) odds ratios sound bigger than they really are, and effect sizes sound smaller, so you should be really careful comparing two studies that report their results differently.
My ratings of the book Likelihood to recommend: 5/5 Educational value: 5/5 Engaging plot: 5/5 Clear & concise writing: 5/5 Suitable for: everyone
Humble Pi is a witty & funny book that could let anyone (re)discover their love for mathematics! Overall, Matt Parker’s book is an appetizing combo of mathematics and comedy – if you want to learn mathematics while having tons of fun, this is one of the best books to start with, regardless of your background or fluency in maths.
Beyond making maths digestibly fun (and funnily digestible), another highlight of the book is how to think about thinking. In other words, the philosophy of thinking – such as how to be rational and how to prevent errors.
I particularly enjoyed the “Swiss cheese” model in thinking about errors: think about each error like a hole in a slice of cheese. And horrible sh*t (disaster) happens when somehow the holes are lined up together and the error falls through slices of cheeses, and lands in the pot of catastrophe. More often than not, a catastrophic consequence is the accumulation of a few errors – seemingly minor errors if we look at them alone – but when added together could bring explosive effects. What this means is instead of focusing too much on achieving 0 errors (which is desirable yet almost always impossible), what is more practical is to focus on improving error-detection that spots an error early – patch the first hole in the first slice of cheese, so that it does not trickle down into the remaining slices.
I would also highly recommend checking out Matt Parker’s YouTube videos: his talks at Google and the Numberphile channel, which features bite-sized videos by various mathematicians on everyday-maths and has 3M+ subscribers to date (April 12th, 2020).
Below I quote some parts of the book that I personally find insightful:
1/ We are used to going from theory to application, though sometimes the reverse happens: the application comes first, and then we discover the underlying theory afterwards. We should not let the joy of discovering the application over-shadow the need to fully understand the theory behind – otherwise, using the tool without really understanding its risks could hit us in the foot.
“There is a common theme in human progress. We make things beyond what we understand, as we always have done. Steam engines worked before we had a theory of thermodynamics; vaccines were developed before we knew how the immune system works; aircraft continue to fly to this day, despite the many gaps in our understanding of aerodynamics. When theory lags behind application, there will always be mathematical surprises lying in wait. The important thing is that we learn from these inevitable mistakes and don’t repeat them.“
2/ Don’t underestimate how little attention the public & institutions could pay to math – and what is most frustrating is not the mistakes themselves (which could be absurdly hilarious), but the lack of respect for mathematical facts or a pursuit of truth.
Matt Parker wrote to the UK government after he discovered that the geometric shape of the football was wrongly painted on signs in the UK (unlike the white hexagons, the black shapes on the ball’s surface should be pentagons instead of hexagons). However, the official response from the UK Department for Transport was: “Changing the design to show accurate geometry is not appropriate in this context.” Matt Parker clearly did not think too highly of the response he got:
“They (the Department of Transport) rejected my request. With a rather dismissive response! They claimed that (1) the correct geometry would be so subtle that it would ‘not be taken in by most drivers’ and (2) it would be so distracting to drivers that it would ‘increase the risk of an incident.’ And I felt that they hadn’t even read the petition properly. Despite my asking for only new signs to be changed, they ended their reply with: ‘Additionally, the public funding required to change every football sign nationally would place an unreasonable financial burden on local authorities.’ So the signs remain incorrect. But at least now I have a framed letter from the UK government saying that they don’t think accurate math is important and they don’t believe street signs should have to follow the laws of geometry.“
3/ While (most rational) people agree that 1 + 1 = 2, people don’t always agree on how the same number should be interpreted. A number ceases to be objective when subjective narratives are at play, hence we should not let our guard down and think an argument is “logical” just because numbers are used.
“It seems that, if the Trump administration couldn’t change the ACA (Affordable Care Act) itself, it was going to try to change how it was interpreted. It’s like trying to adhere to the conditions of a court order by changing your dog’s name to Probation officer.”
“[T]he Trump administration wanted to allow insurance companies to charge their older customers up to 3.49 times as much as younger people, using the argument that 3.49 rounds down to 3. […] They might as well have crossed out thirteen of the twenty-seven constitutional amendments and claimed nothing had changed, provided you rounded to the nearest whole constitution.”
“If there are enough numbers being rounded a tiny amount, even though each individual rounding may be too small to notice, there can be a sizeable cumulative result. The term ‘salami slicing’ is used to refer to a system by which something is gradually removed one tiny unnoticeable piece at a time. Each slice taken off a salami sausage can be so thin that the salami does not look any different, so, repeated enough times, a decent chunk of sausage can be subtly sequestered.”
4/ Precision and accuracy on two concepts with nuanced differences, and it is important to not mix the two. Precision is “the level of details given“, while accuracy is “how true something is“.
5/ Be ware of the word: average. Whenever you hear someone talk about averages, emind yourself of this commentary on the census from the Australian Bureau of Statistics: “While the description of the average Australian may sound quite typical, the fact that no one meets all these criteria shows that the notion of the ‘average’ masks considerable (and growing) diversity in Australia.” I would also add that the notion of the “average” masks how the average person is likely to overrate the concept of averages.
“After the 2011 census, the Australian Bureau of Statistics published who the average Australian was: a thirty-seven year old woman who, among other things, ‘lives with her husband and two children…in a house with three bedrooms and two cars in a suburb of one of Australia’s capital cities.’ And then they discovered that she does not exist. They scoured all the records and no one person matched all the criteria to be truly average.“
6/ Correlation does not mean causation. Just because two things have a high chance of happening at the same time does not mean one caused another. For example, I don’t think the number of math PhDs has any causal relationships with how much cheese people eat.
“For the record, in the US the number of people awarded math PhDs also has an above 90 percent correlation over ten years or more with: uranium stored at nuclear-power plants, money spent on pets, total revenue generated by skiing facilities, and per capita consumption of cheese.“
7/ Finally, this is one of my favorite quotes of the book on what mathematics is: “Mathematicians aren’t people who find math easy; they’re people who enjoy how hard it is.“
I hope this book will rekindle your love for mathematics – or help you find it if you have never fallen in love with it in the first place.