In that case yes, I agree.
Being that precise in the way you reframe the question would definitely lead fewer people to suggest “8” (sadly not zero of course, I’ve met “people”)
Now getting people to be more aware that they need to be that precise in the first place? There be dragons, and it is related to your previous comments on the need to ask better questions etc. That is an interesting subject all on its own.
So let’s pause and look at the study. Reflect a small bit on it.
Which the three questions below (the actual cognitive reflection test, aka CRT) do you think is a similar question to the one of the video. I do not think any are at all similar, or as annoying.
What was the method used to get the results?
Biggest cohort was University of Michigan Ann Arbor and there 80% failed to get all three correct. At MIT, smaller n, 52% failed to get all three correct.
It is certainly possible possible to jump to a conclusion about those findings, almost intuitively, but that intuitive answer, that a majority of students at MIT, who have excelled on much more complicated and confusing math questions in testing situations for years to be able to just get into that school, would not be able to answer those question because they approach math intuitively and without reflection, is absurd upon reflection.
Not sure about you, but even as a budget tight college student decades before the time of that study, eight bucks, told “answer as many as you can” of what I am told is a timed 45 minute test, me? I’m not reflecting on a single item, I’m flying through that as fast as I can giving the first response that pops in my mind, motivated to speed answer, both by not wanting to have to waste the full 45 minutes, and being primed by the phrasing that implies 45 minutes may not be enough time to answer all the questions.
In fact at that point in my life my fairly extensive timed test taking experience would have primed me to go fast: first goal is to leave none unanswered and my first impulse was frequently no worse the one I got by overthinking; double back to ones you have less confidence on right off, if time allows and the test result matters anyway. DON’T ever get bogged down on a single question and then not have time to answer all of them. Save critical reflection for, well everything else but a timed test.
There ARE some meaningful questions associated with the concept of “cognitive reflection” … (how much much effort goes into “reflecting” based on individual backgrounds and skill sets, and what motivates approaching a question critically or not, etc.) … the “most annoying question” and the presentation in that video fail to illustrate them is all.
I think the first two are similar to, but much more simply expressed than the junction question. As they are more simply expressed there is less room for ambiguity and as such are likely to feel like less of a “trick” than the junction question. The junction question however still feels to me to be a better example of a messy, awkward situation that the real world throws up that would indeed be susceptible to the cognitive error in question.
The other three are very simple, very much ring-fenced and as such much less open to interpretations and definitional ambiguity. I see clearly why the junction question is considered to be “annoying” in comparison. I still think it has value, it raises other questions beyond just the simple “d’oh!” factor of the other three.
Really? The first two are calculable math problems, just ones that answering correctly takes a moment or so to think about as algebra more than just adding or dividing.
The junction has no right answer and the wrong answer relies on creating confusion over the task. Not a straightforward math question at all really but pretending to be one. It’s more a stumper gotcha, misleading respondents into an incorrect frame.
Was it also asked as part of a 45 minute test with many items?
They are alike in that for all of them a simple, but wrong, solution presents itself and it is a solution that many people answering do indeed choose.
In essence none of them are about maths at all.
no idea, I’m working to find out.
They present as simpler math problems with simpler solutions than they are, but they are still straightforward math problems with a definite single value right answer. If you are familiar with exponential functions the lily pad one has a quick and obvious single answer. If that is not part of your everyday thought process, then it’s a thinker.
The traffic one instead depends on its confusing set up and presenting the problem in a way that primes for applying what ratios are equivalent as the question. The “trick” to it is parsing out what the question actually is, and realizing that answering the question requires placing a value on lives lost or disabled.
Not at all of the same type.
On reflection can you understand why being part of a timed test battery, answer as many as you can, being done to get eight bucks, biases the results?
Rather than simply jumping to the easy “out” that the simple numbers offer? That to me is where the similarity lies and it has enough similarity to act as starting point for the rest of the video.
You disagree, that’s fine.
I can certainly see potential for that time pressure to lead to quick answering and people being more prone to taking cognitive shortcuts
I think that the biggest hang-up with the traffic-junction question is that people are conditioned to believe that math questions always have a clear, well-defined answer, and so we intuitively reject any approach that won’t give a well-defined answer. The equivalent total damages interpretation won’t give a definite answer, but the ratio interpretation will, therefore the question must mean the ratio interpretation. But this is a problem, because it’s only in math textbooks that questions work that way: Real-world problems always include some degree of assumptions, estimates, and filled-in data. We [i]should[/b] be training ourselves and the next generation to solve messy real-world questions.
To those who are saying that the problem is just the phrasing of the question, how would you phrase it? I’m not sure there is a better way.
Yes, but given time, would you have doubled back to those questions, or would you have said, “yeah, the balls $0.10, no need to look again”? The whole point is that lots of people will be confident in their answer, and be wrong.
The whole question in the OP is:
What is z?
All of the words are misdirection to distract from the fact there is no numerical solution. Lots of people latch onto the easy pattern and say z=8, which is the point.
I’m curious how many people would get the bat and ball answer correct if the questions were put as
What is y?
I’m sure lots of people would still have trouble, but would they latch onto y=0.10 as the wrong answer?
The phrasing is poor, but the bigger problem is that this setup involves the introduction of a bizarre ratio of major to minor accidents. It is left completely unexplained, it is not even commented on. But we are simply supposed to ignore this elephant in the room to get the answer “right”. And this is absolutely NOT what a rational person should do when faced with a real life problem.
That element is certainly not present in the $1.10 problem or the widget problem or the lily pad problem.
But this needs qualifying, see @DSeid 's post upthread about the context in which these questions are posed. I think what you say is true, but I see no valid inference from this that this is necessarily the way people would respond to a real life question - this is the way we respond to a sloppy video on the internet, or a 45-minute test that undergraduates are being paid $8 to complete quickly.
Facts not in evidence.
It would interesting if the questions were introduced with:
“Take as much time as you need. You will be paid $10 for each correct answer and penalized $20 for each incorrect answer.”
Now give me the CRT seeing for the first time. Just the three.
As part of a big timed test I’d have answered the bats one and the lily pad one correctly as the immediate response. No reflection needed. The widget one would throw me and I would put something immediate guess down and come back maybe after I answered the rest.
Under the conditions of wrong answers penalized more than right ones rewarded? I’d really reflect on the widget one until I was sure.
Point is that context matters lots. What contexts and in what ways are interesting questions. When do we default to shortcuts, heuristics? When do they fail us and when are they useful time savers?How do things like our past training, experience, contextual cost of failure, extant beliefs, so on, impact our tendency to stop and question using the shortcut, to double take?
There’s an illustration of medical diagnoses making that gets made lots that is pertinent: Aunt Minnie.
You see someone in the distance wearing what your Aunt Minnie usually wears, near her house, built like Aunt Minnie, walking her walk. You jump to it’s Aunt Minnie and speed up to say hello. At some point though you notice something that jerks you that that woman is not Aunt Minnie at.
Diagnostically most colds are colds, most back pain with fever in flu season is flu related. You don’t evaluate every cold for sepsis or look for spinal epidural abscess for every back pain with fever. Straightforward diagnoses are made without much reflection. You default first to recognizing a pattern but then with more experience to recognizing when that the pattern doesn’t fit and pulling back to reflect and reconsider.
Using heuristics is not bad thinking. It is what makes us efficient Being able to recognize when the heuristic is inappropriate and anchoring you to a false conclusion is a key skill.
I’d certainly be interested in learning why this bizarre ratio was chosen for the problem, and whether its originators consider this a bug or a feature.
Apart from the fact that a lily pond doubling in size every day for 48 days would cover 20% of the earth’s surface. And where are you buying a baseball and bat for $1.10?
Both of those are bizarre and would require some questioning in the real world.
You are reaching here - they are not bizarre in the same way.
We have no problem with understanding a model where something doubles every day, and spotting that the questioner may not have taken note of just how big something gets in 48 days of doubling doesn’t affect what we think the model is supposed to be.
We have no problem understanding that the fact that it is specifically a baseball bat is irrelevant, and that the question might have been written many years ago and never updated for inflation. (Not to mention that it doesn’t in fact say baseball bat, it just says bat and ball - could be any cheap toy.)
But a bizarre ratio of major to minor accidents is something that prominently begs explanation when it is essential to modeling the problem that you evaluate the relative significance of each.
What’s so bizarre about the ratio? There are plenty of plausible real-world scenarios where that ratio would be reasonable. Like, if this is an 80 MPH road: It’s really, really hard to have a minor accident at 80 MPH.
I wondered about that and tried, briefly and unsuccessfully, to find real-life statistics for major vs. minor accident rates on high-speed highways.
Even high speed multi-lane highways have more minor accidents than major (most accidents happen when there are queues of traffic, so don’t happen at 80mph). Also it sees 2000 major accidents! Even the most insanely dangerous junction wouldn’t see anything like that IRL.
Disagree - most cars on the road these days are extremely good at dynamically preventing (or if not, mitigating) bad accidents, and in the even that fails, their safety cells are astonishingly good.
In the UK, our motorways (legal limit: 70mph, though a lot of vehicles are travelling at 80mph and a few, faster still) are statistically the safest roads per passenger mile - yes, some of the accidents are horrendous but they’re the ones that (sometimes) make the news. The vast majority will see those involved walk away with minor or no injuries (e.g. spinning out and hitting a crash barrier). Albeit the disruption can be major, in terms of hours lost to traffic hold-ups while someone literally picks up the pieces.
There is also the fact that I doubt there are any roads in the world with frequent speeds of 80mph that actually have junctions - at least, junctions that don’t involve on/off ramps.
A 250:1 ratio of major:minor accidents is well outside the realm of normal human experience.
Yeah Ihose are genuine example of a “cognitive reflection” problem (the first one especially has an obvious, wrong, answer. I definitely thought “1$ and 10c” when I read it.) The junction question as well as being bad, isn’t really a “cognitive reflection” problem. Even if you accept that 8 cannot be the right answer the only reason it appears right is because the question is phrased badly, it has nothing to do with “cognitive reflection”