I wouldn’t make that conclusion in such general terms. It is definitely a factor (and one that the abstract you pointed to agrees with) but how big the effect is would be up for debate.
Again, I don’t think I’d phrase it like that, I wouldn’t go into more than/less than average at all.
I’d more state it this way, that someone at risk of getting it wrong, without any reflection would be less likely to do so with greater awareness of when and how to reflect on the question and their answers. Heck, even those getting it wrong after reflection could benefit from better tools for doing so.
But I do agree with this (as does the abstract you quoted)
I’m not so sure about that. Given the sort of correct/incorrect ratios seen in the original study that doesn’t ring true to me. I’d be interested to see your calculations and assumptions because I don’t think we can’t make that sort of a calculation without access to the full paper that you referenced.
I’m not saying you are wrong, just that the calculation isn’t instinctively obvious to me, I think we have information missing here.
Maybe. And there is definitely an important distinction between that and making an effort to ask a question clearly. When a large fraction of your target audience understands your question differently than what you meant to communicate, then the problem is the questioner, not the respondents. Unless of course the intent was to be confusing and misleading. It is easy to be confusing; harder to be clear.
A bit ironic eh?
Sure let us go step by step in a way that should make it very clear, if it isn’t intuitive based on the higher percent of correct answering people were not reflecting than those who answered incorrectly.
Let’s use the 2 to 1 incorrect to correct but it could be any ratio. 1000 subjects say.
Do the box method. Same sort of box as in the video. Top is correct or incorrect. Side in reflected or not.
Of the thousand 333 were correct and 667 incorrect. Of the correct 23% reflected (77) and 77% (256) did not. Write those numbers in that first column.
Of the thousand 667 were incorrect. Of them 39% (260) reflected and 61% (407) did not. Write those. Numbers in the second column.
Now go across.
Of the reflective respondents 260/(260+77) or 77% were incorrect.
Of not reflective respondents 407/(407+256) or 61% were incorrect.
Of all 67% were incorrect. More often incorrect in the reflective group than the not reflecting one.
Knowing whether or not they reflected helps predict a greater probability of an incorrect response.
Let’s just say it was intended and leave it at that.
See, this is the sort of fun we all need on a rainy Tuesday evening. It is at this point in a discussion that I normally grab a coffee and a white-board.
I think I see what you are saying.
With the numbers you gave, If you pulled two random people out of that sample of 1000 and called them A and B. (like the junctions ) And all you knew about them was that A was reflective and B was non-reflective, then you should be more confident of predicting that A was “incorrect” than you are of B being “incorrect”.
Whereas the premise of cognitive reflection as a means of reducing error would predict the opposite. One should expect the reflective one of the pair is more likely correct.
This is of course correlation and causation needs to be attributed cautiously.
Actual numbers not far off from assumed 2 to 1 - 28% correct; 60% “intuitive incorrect”; 8% other incorrect; and 4% gave up.
Read away if interested.
I am skeptical of the underlying assumption of “cognitive reflection” - an assumption that the use of heuristics is automatically a poorer approach than greater cognitive reflection and second guessing. I would argue that even IF cognitive reflection resulted in fewer errors, each can be of greater utility in different circumstances, and a while commonly a bit of stepping back is worthwhile, heuristics are often valuable, allowing for, not as described in the article, “miserly” use of brain resources, but efficient use of those resources. There is alway a balance between accuracy and speed, and recognizing that there is a mismatch to the general pattern that the heuristic works for does not necessarily require reflection.
Thanks for the full link, it is a fascinating read.
There’s no simplistic answer for any of this, just the associated papers mentioned in your link could keep a person busy for months and there are plenty that back up the benefits of reflection and also plenty (the cited paper included) that can be summarised as “yes…but, it isn’t that simple and there are other factors at work here and some strange results that warrant further investigation” Which I’m sure others will go on to do.
A really productive discussion though, thanks for the engagement.
Also, without being too meta about it and regardless of the absolute power of reflection and the CRT, I think there have been plenty of exchanges in this thread that practically show the benefit of being open to reframing, restating and clarification as a means to better understanding.
That’s an interesting take. Thinking about it, that also aligns with some of the best practices raised previously in the thread.
Stopping to ask the right questions, and by doing so clarify, specify, restate, narrow down, focus in etc. that can be seen as a means of pruning away the fluff and so allow us to deal just with the real question and the relevant data that we need to do so. Giving a clearer view on which those heuristics can operate.
I think that is reasonable bottom line. Heuristics often serve us well if we apply the right ones in the right circumstances. Our brains use them for good reasons.
Of course when our brains use various processing tricks in perception we can have perceptual illusions which make use think we are seeing things that aren’t there. Study of those illusions informs about how our brains work. There are what get called cognitive illusions as well.
They can make for more annoying math problems!
One at a time. Try to answer before reading other responses.
There are four cards showing the signs or symbols A, K, 4, and 7 on the front side of the cards.
Rule:
If there is a vowel on one side of the card, then there is an even number on the other side.
Which card(s) must be turned over to check whether the rule applies?
If the intent of the question is to prove the purported rule is true for all four cards then you must (potentially) turn over all 4. The first card you turn which violates the rule means the rule is disproven and you’re done. If you turn over all 4 finding no counterexample, the rule is proven.
If the question really asks “Determine which of the 4 cards (if any) this rule applies to.” you now always need to turn over all 4. in effect separating them into two piles: those that follow the rule and those that don’t. One of the two piles might be empty.
Said another way, the question is ambiguous.
As to the folks which suggest turning two cards I’ll point out the proposed rule doesn’t say “front vs back”; it’s just “other side”. Huge difference.
ETA: Ah, I messed up. The question didn’t specify that there’s always a letter on one side and a number on the other. So you do need to turn over K. But you don’t need to turn over 4.
