I recall once hearing, “the higher you go in school the worse you do on multiple choice tests” with the rationale being, “it’s not always A or B, could be C, and I could probably argue D, as well.”
Has anyone else ever heard this? If so, a citation? If not, does it make sense to you?
I haven’t heard it and I don’t think it’s true for me.
No, no, and yes. As I progress farther in my training, I am much more attuned to evaluating questions - are the terms defined? Does it account for all the data? What else is going on? Multiple choice questions are nearly always too vague in some way.
I always have had problems like this with multiple choice, so I wonder if cause and effect may not always be straight forward.
I have never heard this but it sounds like a dry academic joke. The more advanced your degree, the less distinct are the issues you are trained to study. When you’re a freshman in economics, and they tell you supply goes up, and ask what happens to price, the answer can only be that price goes down. But if you’re a PhD candidate and they give you the same question, you now understand that economics is so complex (that is, it is never the case that the only thing that changes is supply going up) that there is no one “correct” answer. (IANAE, I pulled this example out of my ass so don’t take it too literally.) Instead you are expected to perform analysis and justify your answer.
(My graduate economics prof said that you learn the same stuff when you get a PhD as when you’re a freshman, it just takes that long before you start to believe it.)
I think it’s more of a metaphorical hump than a wall. Most tests ask for the best answer.
A beginner might say “this is the answer” - he sees everything as being black and white. With more learning, a person might start thinking “all of these answers are possible depending on the circumstances” - he sees everything as gray. But with more experience, a person will come to realize “all of these answers are possible but this one is the best” - he’s learned to tell the difference between shades of gray.
I think it could also highlight a research/cram distinction. A good philosophy student might memorize before a test that Plato died at approximately 80 years old, was a disciple of Socrates, and presented the argument of forms. A good Phd philosophy student might not particularly care, and dedicate himself to analyzing the theory of forms how it influenced the world.
This is kind of a bad comparison on my part (in this case, knowing the social standing and connections of the person involved would help), but I’ve met no small number of researchers in, say, math, who say “yeah, I don’t have trig memorized, I just look it up when I need it.” 98% of my high-school trignometry tests? Spitting out identities, “quick, whats sin(pi/2)?” and other semi-meaningless bits of mathematical trivia. Similarly with professors of history that I’ve talked to.
That said, “multiple choice” isn’t really a problem. I’ve had well written multiple choice tests, and terrible ones. There are some tests I think are more geared to trivia junkies and some more geared towards the analytically minded, but I think the majority are definitely in the “trivia junkie” category – especially since the sheer volume of questions on most multiple choice tests is geared towards quick recall as opposed to critical thinking.
ETA: Also, I think the “reading comprehension” portions of standardized tests can suck it. Maybe I just legitimately don’t fall into the category of “mastering English,” but I could argue the answer to “what is the tone of this passage” from dusk til dawn and not get tired on some of the questions. Granted, I can usually guess what they intended to be the answer even if I don’t agree.
Perhaps a more legitimate comparison would be “a decent philosophy student has memorized facts about Plato, a good PhD student knows how to argue with Plato.” Which doesn’t necessarily entail knowing facts like “he was about 80 when he dies.”
I think the last multiple choice test I ever took was the GRE (51 years ago) and I sent through it so quickly that I didn’t have time to think about the questions. I did pretty well too.
Agreed. Also, there’s “all of these are possible, but knowing the teacher’s way of thinking (or degree of incompetence) this one is the most likely”.
There’s also “None of these are especially correct, but one of them is the least wrong.”
And I’ve seen personality/psych tests for Government jobs that contain “Have you stopped beating your wife?”-type questions with answers ranging from “Not true” to “Very true”. :smack:
I doubt if it’s true in practice, notwithstanding the points made above about the inadequacy of the multiple choice format for testing higher-level knowledge. An intelligent, highly-educated person who has dealt with many, many exams and exam questions is not in reality going to be stumped by a multiple-choice question because he/she is aware of some imprecision in the phrasing of the question or because there is some overlap or incomplete coverage in the available answers.
If anything, the exam-experienced person will be more skilled than a more exam-naive question at figuring out and playing by the implicit “rules” and choosing the answer that the examiner is looking for.
Speaking of odd multiple-choice questions, I had a friend who was given an aptitude test that included the question:
5) How well do you feel you understand this question?
with multiple choice options that ranged from “Very well” to “I do not understand it at all”.
What I hate about multiple choice is that say A is the correct answer, but B is also right but not the best answer. C is vaguely true, but contains something that very wrong. D is flat wrong.
If I put B, and dumbfuck next to me puts D, we both get marked incorrect and get zero points, even though I was on the right process and he completely blew it. Perhaps a 4, 3, 2, 1 point allocation would be better.
Also, most multiple choice questions are poorly written because the writer didn’t see the ambiguity, but others see it.
One of the things I dislike are those personality tests that say things like:
I would like to date a sorority girl
a) not true at all
b) somewhat false
c) neither true nor false
d) somewhat true
e) completely true
What’s the point here? I wonder when seeing these things if you are expected to realize that the question being asked is not a question of degree any more than being asked the integral of x^2 dx, and so any answer other than a or e demonstrates that you don’t understand English or common sense and logic. Either you want to date a sorority girl or you don’t.
You COULD ask something like:
How much would you like to date a sorority girl?
a) Not at all
b) Not very much (e.g. maybe if the perfect one showed up on my doorstep, but I’m not looking)
c) Meh
d) Very much so (e.g. where are the sorority girls I gotta find one now! <3)
e) I <3 sorority girls so very much that they occupy my every thought
I had a professor in college who was from somewhere in West Africa but taught American history. (Actually, she taught African history because of complicated college politics, but American history was her area of research.) She had a bitch of a time on her citizenship exam because the questions and answers were so simplistic to somebody who had spent decades studying the issues at hand that she worried that she hadn’t picked the answer the testmakers were looking for. (She did pass, though.)
Yes, something similar happened on The Simpsons when Apu was taking the citizenship exam. One of the questions was what the cause of the Civil War was. The answer that the official wanted, of course, was “slavery”, but Apu went into a discussion of all the conflicts facing the US at the time. Of course, in reality, there were other aspects to the Civil War such as states rights and economic pressures relating to the disparity between the industrial North and the agricultural South.
As mentioned in many other places, the key to multiple choice tests is to find the answer that the test maker would consider the correct one. You have to try to think like the test maker (e.g. if the test was written by a 4th grade teacher to be given to 4th graders, then any answer that requires highly complex or convoluted reasoning or requires knowledge that a 4th grader wouldn’t be expected to know is not the right answer in the context of that test, even if your uncle the PhD could put forth an argument as to how some scholars could consider it correct if you superimpose the question on the cultural context of a farmer in the Spring and Autumn period of Chinese history, even though Chinese history or culture is not part of the curriculum being tested and there are no references to it on the test itself.)
Likert scales are different than multiple choice though. Multiple choice has one “definite” answer where Likert is still subjective. When results are scored, different measurements are used.
To what personality type of test are you referring, by the way? Curious.
My experience is that if you’re smarter than the person who drew up the test, the odds increase that you’re going to run into problems.
The people who come up with computer certification exams are … not as good as they might be.
Multiple choice doesn’t really enter into it, except you know that the question is definitely screwed up on a multiple choice test. If it’s a fill in the blank, say, you won’t know unless you later are told the official answer.
Anthony Burgess (Clockwork Orange) told a story of one of his students on a test was given 20 words and asked to split them into 2 groups. The student noticed that half were Latin in origin and half came from Greek. He got it wrong. They were nouns and verbs.
The excuse “you should answer like a regular person” just encourages bad test writing.
I can give an example I had on a test for my credential:
How did he US acquire California?
a) Treaty
b) Annexation
c) Purchase
d) Gift
Extra credit if you know why a and c are both correct. I still don’t know if I got it right.