Which of these sequences of events is more likely?
a) Mexico declares war on the United States
b) In response to growing anti-Hispanic sentiment in the US, a grassroots populist nationalist movement grows in Mexico, with a strong anti-US bias. Russian agitators seize on this movement, and gather compromat on a number of up-and-coming Mexican politicians. Russia picks the one on whom they have the tightest control, and methodically stage political attacks on their primary rivals, until the selected politician is elected president of Mexico. The new president, spurred on by Russian handlers, signs trade agreements to import increased amounts of Russian fossil fuels. Meanwhile, the war in Ukraine spills over into Poland, and thus all of NATO. Russia tells Mexico that, unless Mexico joins in a military alliance against the US, Russia will suddenly cut off all fossil fuel exports to Mexico, on which the Mexican economy has become dependent. Reluctantly, the Mexican government declares that they have no choice, and declares war against the US.
Oh, and the smaller hospital will more often have a 60% boy day, and while I didn’t go through the full calculations, even a woman with a positive screening probably still doesn’t have breast cancer, because the fact that only 1% of women gets breast cancer is a much stronger predictor than the test as described.
I’ve also seen a variant of the card test where instead of letters and numbers, one side of each card shows a person, and the other side shows a beverage, and the rule is that a card with a child on it is not allowed to have a beer on the other side. When the problem is presented in this way, most people do correctly check the card showing the child and the card showing the beer, and don’t bother checking the card showing the Coke. Which proves… something. The study I read that presented it said that people are better at problem-solving when ethical rules are involved, but I think that all it really proves is that people do better with concrete examples than abstract ones.
That example makes me wonder if there’s something else going on. Option A looks different once you read option B: instead of “Mexico declares war on the United States. You don’t know anything else about the scenario.”, it reads more like “Mexico declares war on the United States completely out of the blue, and for no obvious reason.” However unlikely option B is–and probably most readers would also give some wiggle room in the details when it comes to declaring whether B happened or not–it is probably more likely than declaring war for no reason at all.
Of course it is, and that is one of the inputs that we use to come up with 7.5% figure. The point is that it’s not the only input.
You must weigh the benefit of a true positive vs the cost of a false negative AND the cost of a false positive, weighted according to the probability that these things will happen.
You described it as “horribly wrong”.
7.5% is the value of the correct metric here. Of course it requires that we also explain to people why that is the correct metric, following up with an explanation of the benefits that will accrue to them with 7.5% probability to weigh against the negative consequences that will accrue to them with 92.5% probability.
To quote only the test sensitivity of 90% would be far more misleading, since that is the wrong metric for whether someone should take the test.
I think that any headline like this, no matter what the percentage or what it means, is misleading, because there are such different things that “x% accurate” could mean.
I would say, if you give a percentage, specify exactly what that percentage means. Is it the probability that, if you have cancer, the test will catch it? Is it the probability that the test will report accurately whether you have cancer or not? Is it the probability that you have cancer if the test says you do? Or something else? All of these could be quite different.
And if that were the headline I would not complain.
And I absolutely stick by that. Such headlines are often used and they are by turns simplistic, purposefully misleading and sensationalist.
It may be, but it is not the value of “accuracy” as the headline seeks to imply and as it would be understood by the general public.
I find it strange that someone so concerned by the ambiguity of the word “equivalent” has no such problem with an even more egregious and deliberate ambiguity concerning the word “accuracy”
agreed, and it isn’t too cycnical to suggest that headlines on these subjects are so constructed as to promote the most sensationalist possible interpretation in the mind of the reader.
If your argument is simply that it’s possible to write misleading headlines, then that’s just silly. Of course it is.
But when you said that this headline was “horribly wrong”, you were implying that this 7.5% statistic somehow has the potential to generate far more egregiously misleading headlines than other statistics. I don’t agree, because at least it’s the correct statistic, and of all the possible senses of “accuracy” this is the one that people should care about when they start to think about whether the test makes sense for them. Of course it could be written better, but it’s not “horribly wrong” - at least it’s the correct starting point for understanding the issue.
If people do not know how to look at the risk reward of taking a test, and they are unwilling or unable to read beyond a brief headline to learn, then no headline should ever say anything more than:
“Trust your doctor’s advice on whether to take this test.”
In fact, I think if research showed that it was beneficial on balance to screen using the test, this would make a great headline:
This cancer test is only 7.5% accurate. Here’s why you should still take it.
Because that will draw people in to read further and understand the explanation of the the cost-benefit analysis, while also preparing them for the fact that if they test positive, they should not panic, because it’s 92.5% likely to be a false positive.
It isn’t an argument it is a statement of fact. The press misrepresent stats all the time in a way that is plausibly true, shot through with purposeful ambiguity and designed to
present a sensationalised and misleading first impression (which often ends up being the only impression for some people)
That was not what I was intending to imply. I can imagine worse examples. Had I meant what you inferred I’d have probably used a phrase like “couldn’t be more wrong”.
Heck, I even stated up front that…
But if someone read that headline and thought (quite reasonably) that the cancer test in question only had a 7.5% of detecting their cancer, then I stand by my qualitative assessment that it is horribly wrong.
The fact that it is not written better, the fact that it is purposefully written so as to be misleading and sensationalist is what makes it so horribly wrong for me.
But I feel we don’t disagree that much at all on the detail of what a good article would be but rather on my turn of phrase for this headline. If “horribly wrong” is too much for you then substitute purposefully misleading.
it is better, but it still suggests the test is highly unlikely to detect your cancer. if clarity is what we are looking for then I think this is more honest and accurate.
“Screening test detects 80% of cancer cases, here are the pros and cons”
If all you care about is misleading headlines in general, why did you jump on this particular statistic the first time you encountered it to construct one? Your strong implication was that this is an inherently misleading statistic. I think the problem is that you don’t understand that this is the correct statistic for the purpose of deciding whether to take a test.
Why not? it is a very good example of a statistic just begging for mis-use by the media.
I did also raise the issue earlier in thread about the media misrepresentation of relative v absolute risk for cancer. It is a common tactic.
There is nothing wrong with the 7.5% figure when placed within the correct context. To write a headline that suggests a specific test is far less accurate than it actually is, that is definitely wrong.
I worked in clinical trials for many years, side by side with those designing and administering them in the labs and in the field.
I fully understand the need to give people all the information required to make an informed decision, so the patronising tone is not welcome.
I also understand that to purposefully misrepresent the accuracy of a test or efficacy of a treatment in the egregious way of my hypothetical headline would be unthinkable. Perhaps that’s why it annoys me so much when it happens.
Then all I can say is that it’s strange that you were unfamiliar with the standard textbook case of sensitivity vs specificity when prevalence is low, and that you still don’t seem to get the point.
If the cost of a screening is small, then the 80% figure is irrelevant in itself (unless you are comparing two alternative tests for the same disease, of course). It means 80% true positive vs 20% false negative. But false negatives are irrelevant. If you have a false negative, then your state of ignorance is unchanged - you are no better or worse off than you were before the screening. What you need to be concerned with is the benefit of a true positive vs the cost of a false positive. A false positive costs more than just the small cost of the screening, because it usually means more invasive and costly follow-up tests.
No test’s accuracy can be honestly described by a single number, and so any attempt to describe a test’s accuracy with a single number is wrong. You always need to give at least both the sensitivity and the specificity, or some other pair of numbers that conveys equivalent information. And the true prevalence in the population is also often relevant (but not a statement about the test itself).
But it will often be the case that the cost of a screening is small or negligible. As men of a certain age will probably know, there has been back and forth on the DRE prostate exam that takes a few seconds as part of a routine physical. It costs nothing, there’s no risk, so why not just do it? When the cost of the screening is small, both false negatives and true negatives are irrelevant, since with a negative result you do nothing more, and if it happens to be a false negative your state of ignorance is unchanged. All that matters is the consequences of true positives and false positives that do prompt further action.
No result “leaves your ignorance unchanged”. If you have a test with a very high sensitivity (even if it has low specificity), then a negative result takes you from “probably don’t have cancer” to “definitely don’t have cancer”. And even with a more plausible level of sensitivity, a negative result increases your confidence in your lack of cancer.