Tests- Truth or UL?

Factual Questions
1

In the thread “Columbus and the Eclipse: truth or UL?”, Fretful Porcupine stated that he was looking for “nuggets of misinformation in the media”.

I had a couple to state, but I suddenly realized that I didn’t have sources, and couldn’t actually prove my allegations. So rather than look like a fool in that thread, I’m starting a new thread to look like a fool in.

So please tell me if you’ve heard about either of these stories (both of them came out in the late '80’s), and whether I’m remembering them incorrectly.

#1: The media goes off on a tear regarding a new survey which shows that well over half of American students can’t find the United States on a map. The country seizes up with apoplexy over student unintelligence. Eventually, someone points out that 98% of students identified the continental U.S. correctly; but because most students forgot about Alaska and Hawaii, they got the test ‘wrong’.

#2: I remember reading in Newsweek an article regarding how German and Japanese fourth-graders performed much better on a certain standards or reading and math test than American fourth-graders. The article raved (as many articles about education do) about how terrible American education was and how it wasn’t even close to giving American students what Japanese and German students were getting. Then, in the next-to-last paragraph, it mentions that Japan and Germany choose their best students to take this particular test, and the U.S. chooses a representational sample of students, and that this might have something to do with American poor performance (well, no duh!).

So am I hallucinating, are these Urban Legends, or are these true examples of media scare-mongering regarding education?

JMCJ

Curmudgeon Of The Day Winner, 1/19/00
As Selected by RTFirefly

2

I remember reading an article (Reader’s Digest?) about some famous survery claimed that 1 in 3 children are physically abused. One of the questions was “have you ever been hit by a sibling?” and if they answered yes then they counted as being “abused”.

3

I think your premise is correct, John that most of these statistics say more about the speaker than they do about the subjects. As your examples have shown it is very easy to bias a survey to get the answer you want.

IIRC, the DNC recently had an online poll that asked whether you preferred the Republicans’ fiscally disastrous tax cut or Al Gore’s wonderful plan to save social security, or something to that effect. Some wayward Republican found the website and spread the word and the poll started coming in strongly in favor of fiscal disaster – Rush Limbaugh got ahold of it and the numbers went through the roof. If not for that happy accident the DNC would have proudly published the results of their survey as a strong endorsement for their policies.

Let me hasten to add that I’m sure the RNC is just as capable of generating statistics in their favor.

I guess that’s why they still hold elections.

That’ll do, pig. That’ll do.

4

Well, about all I do know is that statistics are bullshit for the most part. You can get statistics to back up anything. I can recall reading ridiculous stuff like 3 out of 5 Americans prefer white socks to colored socks. Yeah right. I think the tests thing is very skewed too. America tests kids from all strata of socio-economic status, and all abilities. Whereas other countries test their best and brightest. That screws up the test results a lot. However, the education system in America does suck. We should pay teachers more. A LOT more. Noonch.

“And on the eighth day, God Created beer
to prevent the Irish from taking over
the Earth.”
~SNOOGANS~

5

Lies, damn lies, and … :wink:
Having said that, you should always be skeptical about media reports. Media isn’t in the business of telling you the truth about something, it’s in the business of reporting something to you so that you’ll buy the media, and the advertisements that go with it. It is through selling advertisements that media make money (radio and free television are the most radical examples of this). Unfortunately, this means that the reports you read don’t have to go into detail about all the facts relevant to the message being conveyed.

Keep this in mind when reading two particular types of reports: reports of studies or surveys, and reports of court decisions.

6

You must remember the Golden Rule: Statistics lie and liars use stats. You can fudge and twist the numbers in any fashion you wish to “prove” your point. (Case in point - see my signature line below.)

I’d rather have a bottle in front of me than a frontal lobotomy - Hawkeye 4077th

7

Hey, what the f—? It’s using my old sig line! I changed that long ago! Perhaps this is a quirk since the webpage was updated!!!

It should read:
98% of people think they’re above average.

It’s Murphey’s “Law and Order”

8

[q]98% of people think they’re above average.[/q]

Actually, that doesn’t twist things at all. It is quite possible that 98% of people think that they are above average, just 48% of them are wrong. The error rate could even be higher than 48% if you figure that some above average people have low self esteem and so didn’t count themselves as above average.

For that matter, all of those people could be correct, depending on your definition of average. An example:

Ask 100 people what their annual salary is. 2 answer $12000, 98 answer $40000. Average (mean) annual salary: $39,440. 98% of the sample has above average income. Conversely, average (median) annual salary: $40000, so no one has above average income.

An (hypothetical) example of twisting: An anti-abortion group commisions a poll. The poll question is “Do you believe it should be legal to kill minor children?” Let’s say they get responses of 99% No and 1% Undecided/No Comment. Since they consider abortion and child murder to be synonomous, they can issue a press release saying that their recent poll shows that 99% of the public opposes abortion.

More insidious example: Same poll as above. However, the group knows that abortion is a contentious issue, and that few people would take their press release seriously. So instead they issue a release stating that their recent poll shows that over 73% of the public opposes abortion. Just as valid (from their veiwpoint) a statement of the poll results, low enough to be believable, high enough to constitute a 3:1 majority of the population. A far more effective tool for influencing policy makers.

However, there are lots of good, meaningful statistics out there. They just usually come from people who “want to know” rather than people who have a particular position to support.

Food for thought,
Scott

PS I don’t mean to take either side in the abortion debate with this example. It was just easy to compose a hideously biased leading question.

9

Another example:

“1 in every 10 people is a homosexual”

Apparently the researchers in the study that made this conclusion had two problems:
1.) They broadly defined being a homosexual as simply admitting to ever having participated in a “homosexual” act and

2.) a review of their sample population showed a disproportionate share of prison inmates within the sample.

Disclaimer: Just like Doc-miller, I’m not making a judgment statement here, this is just an example I remember of a situation of inaccurate statistical reporting. I’m not interested in starting a debate on homosexuality in the population. :slight_smile:

10

Johnny…I read up a bit on that for another debate (which is a whole 'nother story)…

Anyway, the two “errors” in your comment on the study are themselves urban legends. The reference is to the original Kinsey studies and their updates, and the statistics of “those who tried it” are around 37%, IIRC (and boy, if I don’t RC, somebody is going to fix it! :)); the 10% are those whose *primary orientation is homo-erotic (13.9% of males and 7.4% of females studied are the stats. I recall). Secondarily, while the original Kinsey study had utilized prison inmates, the updated numbers from which the 10% figure were taken did not.

My own hunch is that that “refutation” you cited was put out by people with their own agenda, though I’m open to being proven wrong. (So what am I doing on this board? ;))

11

Unemployment stats are a strong example, too. They only count those receiving unemployment benefits. Once your benefits run out, you’re not even counted. So, how does this measure the true strength of the economy?

So, when the media boasts “unemployment is down”, think twice! :wink: In general, the media loves to apply stats in any way they can to their advantage. So, think twice!

12

I was under the impression that, in order to be “valid statistics”, the question must be unbiased and the sample taken from a random sample of some population (ie welfare poll taken in a gated community). Can these people get away with calling these generated numbers “Stats” or is there some militant math/stats organization getting ready to set things straight?

Is there another term for these self-serving figures?

kylen

13

In my continuing quest to be totally irrelevant, I run all the way back to the OP to say:

the individual in question is actually Fretful Porpentine, as in “made each hair stand on end like the quills of the F.P.” (Bonus points for refernce spotting?)

And I believe FP is actually of the female persuasion.

We now return you to your regularly scheduled (and quite interesting) discussion.

…but when you get blue, and you’ve lost all your dreams, there’s nothing like a campfire and a can of beans!

14

I read that “US students could not find the Continental US on a map” in a Trivial Pursuit Game, I think it was the 80’s edition

15

Kylen, they’re still statistics, even if the sample is biased, or the question is leading, or there are other errors in interpretation. That does not change the fact that they are statistics, just they’re accuracy and reliability.

16

Jinx wrote:
> 98% of people think they’re above average.

and doc_miller wrote:
>Actually, that doesn’t twist things at all. It is quite possible that 98% of people think that
>they are above average, just 48% of them are wrong.

To which I add:
The overwhelming majority of people have more than the average number of legs.

17

That depends on whether you mean mean, median or mode… :wink:

18

Wow, looks like I inspired a whole thread without knowing it. I’m, um, touched. (And thanks, Da Ace, correct on both counts).

My own favorite skewed-statistics story involves a survey of female students which purported to reveal that 25% of them had been raped during their college years. Turned out that the survey included questions like, “Has a man ever given you alcohol or drugs and then had sex with you?” Sheesh, I’m surprised the figure wasn’t more like 90%…

\\| |/
=== '>

19

Uhh, that’s not the way that unemployment statistics are calculated - at least, not those reported by the US government. That’s done via surveys where folks are asked 1) if they’re surrently unemployed and 2) if they’re currently looking for a job. Basicaly, you have to answer “yes” to both questions to count as being unemployed.

I looked up the gory details at the Bureau of Labor Statistics’ web site, and found a description of their survey at http://stats.bls.gov/news.release/empsit.tn.htm . Here’s the relevant paragraph:

“People are classified as unemployed if they meet all of the following criteria: They had no employment during the reference week; they were available for work at that time; and they made specific efforts to find employment sometime during the 4-week period ending with the reference week. Persons laid off from a job and expecting recall need not be looking for work to be counted as unemployed. The unemployment data derived from the household survey in no way depend upon the eligibility for or receipt of unemployment insurance benefits.”

20

Well, Fretful, I don’t know about the examples you cited, but I used to work in market research, so I know a little something about how a lot of surveys are done.

The outfit I worked for did a lot of work for the Pharmaceutical Association of America. A lot of questions would have a phrase such as " for you and your family" tacked onto the end to put an emotional load on it that was obviously designed to draw a favorable response.

My favorite question- “Prescription drugs are worth their price because they improve the lives of many people.”

Do you
A)Strongly agree
B)Somewhat agree
C)Neither agree nor disagree
D)Somewhat disagree
E)Strongly disagree

The responses tended to be A or B- but here’s the twist. The majority of people who take surveys over the phone tend to make 25K a year or above. Which means they can either afford prescriptions, or they most likely have good insurance benefits through their employers, so cost is not so much of an issue for them. Lower income people tend to be undereducated, frequently don’t follow any news outside of watching the newscast on T.V., and so tend not to be aware of many issues beyond their own paycheck. Usually, a “research technician” will terminate an interview with someone after getting several “don’t know” responses in a row, or the respondent will say something like, “I don’t know nothing about the pharmeceutical companies” and hang up,so the sample is definitely skewed toward middle to upper income people. The people who would be most likely, respond unfavorably to the question would not complete the survey, so their opinions would not be recorded.

One survey we did actually filtered people out at the beginning who had generally unfavorable opinions about the pharmeceutical companies.

If I had told anyone any of this while I was working for the market research firm, I would have been fired, and believe me, if you talk, it gets back to the research firm.

The trouble with Sir Launcelot is by the time he comes riding up, you’ve already married King Arthur.