There is a certain type of news item that just really bugs me…the type which reports some objectively true fact that ends up being misleading, intentionally or unintentionally, because the piece doesn’t offer the necessary context for someone to easily understand how to model the information.
There are a set of questions you need to ask of any data, questions like, “What is the denominator?” If a 600 people who received a certain cancer treatment in the last year died (that sounds bad!), out of how many who received that treatment? If 600 out of 100,000 people who got the treatment died, that’s really different from if 600 out of 6,000 people who got the treatment died. The question, “Out of how many?” really matters.
Another of this type of question is, “What about the untreated group (the control group)?” If 600 people died who got the treatment, you want to know about the people who didn’t get the treatment. Were they more likely to die that those who got the treatment? Maybe the rate of death among those who didn’t receive the treatment is 10 times higher than the rate of death among those who do. The absolute number of 600 doesn’t help you much. You need to know relative to what.
The point is, you can’t really understand what the number 600 means without asking for further information and context. The number may be a fact but it doesn’t tell you the truth on its own.
I find that a lot of news items don’t give exactly this kind of context, offering some data point that is true in the sense of being a fact but not really accurate in the sense of helping you to appropriately frame and model the data.
I admit I rant about this a lot! I am sometimes heard around my house yelling at the television, “What’s the denominator?!” And I am about to rant about one example of this type of news piece below.
I would love to have readers share their own examples of these types of news items in the comments for discussion.