Real life examples of biased test questions?

[muldoonthief]

@MobiusStripes

Darn it! I fail reading comprehension!

It’s OK, you probably grew up poor with a terrible home life.

What I don’t get is the first 2 questions are “good” questions, because they depend on what you learn in school (math & geography), but then questions 5 & 6 are unfair, because they depend on what you learn in school, but some kids aren’t good at what they learn in school? So every question should combine math, spatial, artistic, interpersonal, musical, and linguistic knowledge to be fair?

[Left_Hand_of_Dorkness]

@muldoonthief

It’s OK, you probably grew up poor with a terrible home life.

What I don’t get is the first 2 questions are “good” questions, because they depend on what you learn in school (math & geography), but then questions 5 & 6 are unfair, because they depend on what you learn in school, but some kids aren’t good at what they learn in school? So every question should combine math, spatial, artistic, interpersonal, musical, and linguistic knowledge to be fair?

I agree. A good test tests what you taught, and can be used by many different parties:
-The student figures out how well he or she learned the materials, and decides whether to study harder, to ask for help more often, etc.
-The parent figures out how well the child learned the materials, and decides whether to supplement school with tutoring, to change homework schedules, to talk with the teacher, etc.
-The teacher figures out how well students have learned the materials, and decides whether to reteach certain concepts to the whole class or to individuals, to seek additional help or enrichment for particular students, etc.
-Administration figures out how well students have learned the materials, and decides whether to change curricula, to put more resources into particular schools or classrooms, to hire or fire teachers, etc.

But this can only be done if the test maps closely to the curriculum.

If a question can be answered based solely on what’s taught in school, awesome. If it can’t be, it’s a problem.

(This, incidentally, is why one of the hardest things to remediate is vocabulary. If I’m teaching a third-grader who has trouble deciphering unfamiliar words, I have plenty of strategies for teaching how to chunk words, to recognize familiar patterns, to read in context, etc. But if a kid doesn’t know what common words like “hamster” or “ragged” or “disappear” mean, it’s very difficult to remediate.)

[Hilarity_N.Suze]

[QUOTE-website]
A child who’s linguistic skills are weak but whose interpersonal or spatial skills are strong is at a disadvantage.

[/QUOTE]

An adult WHOSE linguistic skills are weak should learn to hire an editor for things posted on the Internet.

[dataguy]

I had a friend who was an elementary school teacher.
I remember once she showed me examples of questions from a standardized test that she thought showed class bias.

The one I remember was something about where do you keep an airplane. The correct answer was in a hangar. I agree with her assessment. Unless you are from an environment where private planes are common, I doubt this is something a typical 3rd grader would know.

[Meyer6]

@dataguy

The one I remember was something about where do you keep an airplane. The correct answer was in a hangar. I agree with her assessment. Unless you are from an environment where private planes are common, I doubt this is something a typical 3rd grader would know.

What if the word ‘hangar’ had been included on vocabulary lists given to the students in class?

[Breccia]

Interesting questions questioned.

wrt the Deuce and a Quarter

**JoeyP ** …we talked about this kind of bias. The question they used was “What is a deuce and a quarter?” and had a few options under it.

seems the question discriminates between middle/upper class and lower class kids…or music/car fanboys from everyonelse…

Ry Cooder’s* The Very Thing That Makes You Rich (Makes Me Poor)*
Well, I put you behind the wheel of a deuce and a quarter, yes I did
Had you living like a rich man’s daughter, yes I did
While you were living high on the hog
You had me down here scuffling like a dog

…
and ummm, this thing about fruits and seeds…

MobiusStripes …The fruit question threw me, though not for the reasons the website stated.
“A plant’s fruit always contains seeds. Choose the item below that’s not a fruit.”

Originally Posted by Meyer6 The fourth one was celery, not cucumber, I think.

Well. where does celery seed come from then? you know, the stuff us Southerners sprinkle on coleslaw and pickle with cucumbers. ans (as far as I know): celery seed does come from the same genus as what we call celery…and that celery IS a variety of celery referred to as celery in the places it’s grown.
Somehow I did ok on IQ and school tests, likely just luck and curiousity.

[TriPolar]

I don’t see how the term ‘bias’ is used coherently in this test. All tests are biased in favor of those who know the answers. Some tests are testing acquired knowledge, and some tests are testing skills. All written tests are going to have vocabulary and comprehension factors in them. And there is a broad category of ‘testability’ that changes the way many people will score on tests independent of the their knowledge and skill level.

I think the concentration on test results without analysis of the reasons for the results is a bigger problem than the kinds of ‘bias’ discussed in these questions.

[Amblydoper]

@SpoilerVirgin

I will never forget a question on a test I took when I spent 8th grade in Boston, having grown up in California. It involved analyzing the sentence, “The umpire called Yaz out at the plate.” I felt that it should have read, "The umpire called ‘Yaz’ out at the place, although I had no idea what ‘Yaz’ meant (I figured maybe a Boston version of ‘yes’) or why the umpire would be yelling it.

I was a very good student, but that question was meaningless without knowing who Carl Yastrzemski is.

I disagree.

“Yaz” was capitalized, and an 8th grader should be able to recognize a proper noun in the middle of a sentence. Regardless of who Yaz is, it is clear that Yaz is an indirect object in that sentence.

[Doug_K]

The worst kind of bias has been discussed by The Onion.
(Spoilered for possibly NSFW language.)

http://www.theonion.com/video/in-the-know-are-tests-biased-against-students-who,17966/

[Zsofia]

Okay, I looked up the famous “regatta” question from the SAT - it’s an analogy:

runner : marathon ::
A) envoy: embassy

B) martyr: massacre

C) oarsman: regatta

D) referee: tournament

E) horse: stable

I think that’s a pretty clear example of a biased question - all the other choices are common words for high school students. Regatta is on the same level of common words for high school students… who go to Exeter.

[Irishman]

questions paraphrased

Question 1: about computing the total race time given partial race time and distance.

Complaint:

Way to go! This question is taken from a seventh-grade standardized test. It’s asking a student to apply knowlede he should have learned in school. However, many of the other questions on standardized tests ask students to utilize information gained at home. Even if a student answered every question related to what he’s learned in school correctly, he still might receive a poor score. This is unfair to the student who works hard at school, but whose home life is lacking in academic stimulus.

So an example completely unrelated to the complaint. That’s helpful.

Question 2: Country located on the same continent as Brazil.

Complaint:

Many questions on standardized tests not only test school smarts but also assess knowledge gained at home. This is unfair to the child who has an unsatisfactory home life. If all the questions on standardized tests assessed a child’s school smarts, it would even the playing field. However, equality transltes into a lesser range of scores. A large range of scores is what test-makers are shooting for.

So even though this is supposedly a fair question, it is in the quiz and then the same complaint about questions coming from home life. Also note the non-supported assertion that test-makers want a large range of scores, with the implication that this is all they care about.

Question 3: A plant’s fruit always contains seeds. Choose the one that isn’t a fruit.
orange, pumpkin, apple, celery.

Complaint:

You’re right, but a child who’s from a more affluent family gets a break on this item. This question is taken from a sixth-grade standardized test and is testing knowledge that a child usually gains at home. A child whose parents can routinely afford to purchase fresh fruits and buy a fresh pumpkin to carve before Halloween is at an advantage.

Wait, children who aren’t from affluent homes don’t know that a pumpkin has seeds? That celery is a stalk?

Question 4: A Saw is used by a ___
fireman, carpenter, politician, bus driver

Complaint:

This question is taken from a fourth-grade standardized test and it’s also too dependent on a child’s background. If a child comes from a well-to-do family that has been able to do some home remodeling, that child will be well-versed in a carpenter’s use of a saw. So once again, the child with the higher socioeconomic status is at the advantage.

You’ve got to be kidding me. The only way to know that a saw is for cutting wood is to do home remodeling? They’re going to think a saw is something a bus driver uses?

Question 5: There are two circles, with three numbers in each. The question asks which two numbers to switch so that the totals in each circle are the same.

Complaint:

However, this question isn’t fair for many students. It’s taken from a seventh-grade standardized test and it assesses a child’s number smarts. People have many different kinds of intelligences – mathematical, musical, linguistic – to name a few. Standardized tests often only asses one or two of the seven intelligences. The child who may be musically or spatially gifted never gets to show his smarts.

Wow. Just wow. Assessing number smarts isn’t fair? What if it’s a math test? Then is it fair?
Also, just because there is one question assessing “number smarts” doesn’t mean there isn’t another question testing spatial relations. I’ve seen those on tests. You know, take a cube and unfold it. Each side is a different color. Then guess which cube is that cube reassembled.

Question 6: Two boxes, with animal shapes. They have the same shapes, but different locations. Which has moved most?

Complaint:

You’re right – you must have a high spatial intelligence. This question is taken from a fourth-grade standardized test and it assesses a child’s picture-smarts. This question isn’t fair to the child whose spatial intelligence is weak but whose linguistic or artistic intelligence may be very strong. People have many different kinds of smarts and standardized tests only assess a select few.

Wait, you just complained that the previous question assessed “number smarts” and wasn’t fair to “spatial smarts”. Now you follow that with a “spatial smarts” question. Do you not see the irony?

Question 7: How to conserve resources.

• Leave lights on even when not needed.
• Wash small loads instead of large loads in washing machine.
• Write on both sides of paper.
• Place used newspapers in the garbage.

Complaint:

You’re right. This question is taken from a sixth-grade standardized test and assesses a child’s word-smarts. In order for a child to answer this question correctly, he needs to know that “conserve resources” means to save them. A child who’s linguistic skills are weak but whose interpersonal or spatial skills are strong is at a disadvantage.

Once again, whining that one question only assesses one aspect of intelligence. As if every question on every test must include words, pictures, numbers, and apparently musical overatures in order to be fair.

[dataguy]

@Meyer6

What if the word ‘hangar’ had been included on vocabulary lists given to the students in class?

You should re-read my post. The TEACHER felt that this was a biased question.

[Lord_Il_Palazzo]

Most of the things I would say have been said by those above, but I just have pile on with how dumb (or at least poorly presented) this quiz’s agenda seems. Apparently, math tests aren’t fair because not everyone is good at math and questions that require reading aren’t fair as you need “word smarts” to understand what the words mean. With all the complaints (particularly late in the test) about how a particular question only tested “number smarts” or “spatial smarts” or whatever, I have to wonder what they wanted the questions to look like to test every kind of smarts with every question.

Come to think of it, referring to intelligence or aptitude in a particular field as “smarts” seems slangy and colloquial for somebody trying to make a serious point.

[ToeJam]

@Irishman

Question 3: A plant’s fruit always contains seeds. Choose the one that isn’t a fruit.
orange, pumpkin, apple, celery.

Complaint:

Wait, children who aren’t from affluent homes don’t know that a pumpkin has seeds? That celery is a stalk?

From my own experiences of working with families/children, I can certainly believe this one. There are families here who never serve their kids a home cooked meal, and where cheap fastfood or similar stuff is the norm for meals rather than fresh veggies. And if said child isn’t particularly a big reader, then yeah- I easily can understand kids having no idea what a pumpkin or a celery even is. At least among the families I’ve seen coming into the office.

[Agent_Foxtrot]

There is certainly an agenda behind the link in the OP…some of these so-called biases are really a stretch… a richer kid who has had their home remodeled is more likely to know a carpenter uses a saw? Why wouldn’t a kid from a blue-collar family know that too? In fact, what sixth grader really think their bus driver uses a saw in his profession?

No, these aren’t biases. By the link’s rationale, standardized testing should be eliminated altogether because any question could be deemed unfair. “1+2=?” is unfair to the kid with dyslexia. “What is the capital of our state?” is unfair to the kid whose family just moved here a month ago. “Thomas Jefferson wrote the __________.” is unfair to the immigrant kid.

Here are some questions that are really biased:
[ol]
[li]Tom can run a mile in 5.25 minutes. Sarah can run a mile in 6.12 minutes. Why is Sarah out running when she should be home making Tom a sandwich?[/li][li]After reading the passage, it can be determined that the Cherokee people refused to emigrate from their homelands in 1852 because they are:[/li]a) stubborn
b) lazy
c) alcoholics
d) both b and c
[li]In your own words, explain why only fags like the color pink.[/li][/ol]

[delphica]

The only example on that test that I would agree with (and I don’t even feel that strongly about it) is the one about the fruits and seeds. I think it’s biased, not only because less affluent households might not routinely have fresh fruit, but people from non-Western cultures might not routinely have that TYPE of fruit either, no matter how affluent they are.

An example of a possibly biased question that I remember from my own high school years was about probability. On the NY State Regents Exam (this is a state-wide test), a common format for probability questions involved decks of cards, like what is the probability you will draw a three OR a red card in some number of draws.

At my high school, there were students from a religious community that prohibited playing cards. (I think maybe they were some variety of Mormon.) So these students had no idea how many threes are in a deck, or that there were two possible colors, or what “face cards” are.

[Irishman]

Okay, I was digging in to that site, and found this article that explains what the issue is a little better.

http://school.familyeducation.com/educational-testing/educational-philosophy/38778.html

In particular,

Standardized aptitude tests predict how well students are likely to perform in some subsequent educational setting. The most common examples are the SAT-I and the ACT both of which attempt to forecast how well high school students will perform in college.
But standardized achievement-test scores are what citizens and school board members rely on when they evaluate a school’s effectiveness. Nationally, five such tests are in use: California Achievement Tests, Comprehensive Tests of Basic Skills, Iowa Tests of Basic Skills, Metropolitan Achievement Tests, and Stanford Achievement Tests.

To sum up, standardized achievement tests do a wonderful job of supplying the evidence needed to make norm-referenced interpretations of students’ knowledge and/or skills in relationship to those of students nationally. The educational usefulness of those interpretations is considerable. Given the size of the content domains to be represented and the limited number of items that the test developers have at their disposal, standardized achievement tests are really quite remarkable. They do what they are supposed to do.
But standardized achievement tests should not be used to evaluate the quality of education. That’s not what they are supposed to do.

The third reason that students’ performances on these tests should not be used to evaluate educational quality is the most compelling. Because student performances on standardized achievement tests are heavily influenced by three causative factors, only one of which is linked to instructional quality, asserting that low or high test scores are caused by the quality of instruction is illogical.

The problem of confounded causation involves three factors that contribute to students’ scores on standardized achievement tests: (1) what’s taught in school, (2) a student’s native intellectual ability, and (3) a student’s out-of-school learning.

What’s taught in school. Some of the items in standardized achievement tests measure the knowledge or skills that students learn in school. In certain subject areas, such as mathematics, children learn in school most of what they know about a subject. Few parents spend much time teaching their children about the intricacies of algebra or how to prove a theorem.

A student’s native intellectual ability. I wish I believed that all children were born with identical intellectual abilities, but I don’t. Some kids were luckier at gene-pool time. Some children, from birth, will find it easier to mess around with mathematics than will others. Some kids, from birth, will have an easier time with verbal matters than will others. If children came into the world having inherited identical intellectual abilities, teachers’ pedagogical problems would be far more simple.

[snip]
These sorts of items, because they tap innate intellectual skills that are not readily modifiable in school, do a wonderful job in spreading out test-takers’ scores. The quest for score variance, coupled with the limitation of having few items to use in assessing students, makes such items appealing to those who construct standardized achievement tests.

But items that primarily measure differences in students’ in-born intellectual abilities obviously do not contribute to valid inferences about “how well children have been taught.” Would we like all children to do well on such “native-smarts” items? Of course we would. But to use such items to arrive at a judgment about educational effectiveness is simply unsound.

Out-of-school learning. The most troubling items on standardized achievement tests assess what students have learned outside of school. Unfortunately, you’ll find more of these items on standardized achievement tests than you’d suspect. If children come from advantaged families and stimulus-rich environments, then they are more apt to succeed on items in standardized achievement test items than will other children whose environments don’t mesh as well with what the tests measure.

[snip]That’s how children’s socioeconomic status gets mixed up with children’s performances on standardized achievement tests. The higher your family’s socioeconomic status is, the more likely you are to do well on a number of the test items you’ll encounter in a such a test.

One of the chief reasons that children’s socioeconomic status is so highly correlated with standardized test scores is that many items on standardized achievement tests really focus on assessing knowledge and/or skills learned outside of school–knowledge and/or skills more likely to be learned in some socioeconomic settings than in others.

Again, you might ask why on earth would standardized achievement test developers place such items on their tests? As usual, the answer is consistent with the dominant measurement mission of those tests, namely, to spread out students’ test scores so that accurate and fine-grained norm-referenced interpretations can be made. Because there is substantial variation in children’s socioeconomic situations, items that reflect such variations are efficient in producing among-student variations in test scores.
You’ve just considered three important factors that can influence students’ scores on standardized achievement tests. One of these factors was directly linked to educational quality. But two factors weren’t.

Reading that article, it at least articulates better what the complaints about standardized testing are. Primarily, the complaint is using a test designed to help determine what kids need more help learning is being used to evaluate the schools’ instruction ability. Secondly, that the kinds of questions used rely on innate ability and outside knowledge to measure individual performance, rather than just classroom instruction. Those techniques help spread test scores, but don’t help evaluate the teaching ability or educational performance of the school.

The specific example given is the seed question, trying to justify how that is influenced by socioeconomic status. Whether or not you agree with the specific example used, at least the concerns become more understandable in this format.

There does appear to be some content missing from that article. It mentions three reasons why not to use these tests, but the second one appears to be truncated. It lists reason 1 and reason 3.

[Infovore]

@Hilarity_N.Suze

[QUOTE=website]
A child who’s linguistic skills are weak but whose interpersonal or spatial skills are strong is at a disadvantage.

An adult WHOSE linguistic skills are weak should learn to hire an editor for things posted on the Internet.
[/QUOTE]

Ha, you beat me to it, and said it almost exactly the way I was going to!

[doreen]

@ToeJam

From my own experiences of working with families/children, I can certainly believe this one. There are families here who never serve their kids a home cooked meal, and where cheap fastfood or similar stuff is the norm for meals rather than fresh veggies. And if said child isn’t particularly a big reader, then yeah- I easily can understand kids having no idea what a pumpkin or a celery even is. At least among the families I’ve seen coming into the office.

I can easily imagine a child here or there who doesn’t know what a pumpkin is, but I can’t imagine that huge numbers of kids of any socioeconomic status have never seen the bags of pumpkin seeds sold right next to the sunflower seeds on the potato chip racks.

And come to think of it , the saw question bothers me for a similar reason.

If a child comes from a well-to-do family that has been able to do some home remodeling, that child will be well-versed in a carpenter’s use of a saw.

Really? Judging from my experience with affluent people,* that kid won’t know what a saw is or what a carpenter is. The lower-class kid on the other hand, might have an uncle who is a carpenter or a father who built a bookcase out of scrap wood.

Of course, these questions should not be used to evaluate the quality of instruction - but that’s because they are usually unrelated to the instruction , not because they are biased. They **might **be appropriate questions in a cooking or shop class

• I was once the only person in a group of five who knew how to change a tire and had the tools in my car to do it. The other four’s response was “call Triple A” . I don’t think it was a coincidence that I was the only one in the group who grew up in a family that couldn’t afford to pay someone to change a tire , repair a wall, fix an electric outlet , etc and they grew up with a toolbox that consisted of “The Yellow Pages and a credit card”

[Filbert]

@Breccia

Well. where does celery seed come from then? you know, the stuff us Southerners sprinkle on coleslaw and pickle with cucumbers. ans (as far as I know): celery seed does come from the same genus as what we call celery…and that celery IS a variety of celery referred to as celery in the places it’s grown.
Somehow I did ok on IQ and school tests, likely just luck and curiousity.

Celery doesn’t grow a ‘fruit’ containing seeds; it just grows seeds without any extra fleshy covering. Same as sunflower seed.

Incidently, I never saw the inside of a pumpkin until I was in my twenties- they were not a commonly available fruit in the UK when I was a kid- or at least not a cheaply available one. Don’t think I had pumpkin seed until about the same age either.

I think you could just as well argue the ‘saw’ question the other way round- you’re more likely to know what one is if you’re poor, and have to fix stuff. My family certainly wasn’t rich when I was growing up, and I damn well knew what one was… Hardly a ‘Pick the correct fork for eating fish’ question.