(Posting this with permission from the moderators).
I am a postgraduate student working on an instrument that measures attitudes on standardized testing. I’ve developed a twenty-one item survey that asks respondents to evaluate their level of agreement with a series of statements. If you have five minutes to give, I would appreciate if you took the time to complete the survey. There are three demographic items at the beginning (asking age, gender, and level of education), but no identifying information is otherwise requested. Anybody from the United States is welcome to participate.
This is very early in the development of the scale, and honestly, I wouldn’t even consider this a pilot. This is likely the first draft in a long series of drafts, so there’s still a lot of work to be done. The more responses I get, the easier it is to evaluate the strength of the items and the scale.
I took the survey. Personally, I think that they’re the worst option available, except for all the other options. We need to have some standard objective measures of students and, by extension, of educators. But the implementation often leaves much to be desired, and it’s really hard to design tests, and methods for interpreting the test results, that actually give you what you want.
I’m not at all convinced we need to have objective measures. I mean, sure, it’d be nice to have them. But when we upgrade “nice” to a necessity, it means we’re willing to put up with a mountain of bullshit in order to meet the supposed need.
Objective measures of students and teachers are good if they are precise and accurate and not subject to “teaching to the measure.” I have not so far encountered such a measure.
Subjective measures may not be the gold standard, but they come with so much less bullshit that they are, for the time being, superior.
It’s worth noting that a subjective measure doesn’t have to be arbitrary. My principal observes me fairly regularly, discusses my teaching with me often, and writes up my evaluation once a year. That’s not “objective”, but it’s not meaningless. Could she be biased? Might she have favorites? Sure. But there are problems with all systems.
As for the questions on the survey–what tests and what traits? My kids take state exams, the SAT, various AP exams, the ACT, the OECD (a version of the PISA), the AIME, SAT subject tests, the PSAT, district written final exams . . . etc., etc. All qualify, to some extent, as “standardized”, all vary tremendously. Most are pretty useful tools if you understand exactly what they are measuring and don’t try to extrapolate much beyond that. For each of them, I’d have different answers for all the questions on the survey.
You may want to refine your target group definition, “from the US” is one of those things which on one hand everybody knows what it means and on the other maybe not. You know, like “natural born citizen”.
Would you be interested only in opinions of US citizens or nationals currently living in the US, or would other groups such as foreigners living in the US or Americans currently living abroad be of interest as well?
Agree. I actually write test items for a standardized testing system (which determines federal educational funding, but doesn’t affect teacher evaluations). I would have to say that most of those questions are unanswerable because they don’t specify which tests are being referred to.
Honestly, Tarwater, I think if you really want to collect data of any value, you should try to build in some kind indication of the particular tests in questions, or at least a way for the respondent to indicate which tests he or she is familiar with.
I had written this into the introduction for an earlier administration, but not here, for some reason. Thanks to guizot, Nava, and Manda JO for bringing this stuff to my attention:
You don’t have to think about a specific instrument for the survey. Just think generally about your experience with the phenomena of standardized testing. If you’ve never taken a test yourself, that’s fine. If you’re a teacher and you have to administer 3-4 tests a year, that’s fine, too.
Somebody may answer the survey with a specific test in mind, somebody else may not. For my purposes, the data is treated exactly the same. Insofar that I’ve asked for respondents from the United States, that’s only to capture attitudes that are uniquely informed by our culture of testing. If you’re not technically from here, but you’ve lived here long enough to have an opinion, then you’re welcome to share it.
It’s not so much that I’m worried about favoritism, as that a subjective evaluator just might not be able to actually recognize good teaching. I was at a professional-development event recently where the presenter made a point of telling the students what good work they were doing, for instance, when they weren’t: They were all just sitting there dumbfounded holding their pencils over their papers, with no clue of what they were supposed to be doing. And even if they had been working hard, were they learning in the process? Well, what does it mean for them to be learning? How can you tell? Maybe some principals have the knack for telling who’s learning well, but that doesn’t change the problem, it just shifts it up a level: Now, how do you tell which principals those are? At some point, you need objective data to fall back on, to calibrate all the subjective data.
I experimented with multiple choice tests (they are SO much easier to mark) and found them totally unsatisfactory. So I am negative on the whole idea.
Another thing: if they measure innate ability how do you explain the fact that when I took the college boards I was above average, but not that far above (under 700) in math and slightly above average (under 600) in English, but when I took that grad records four years later I was offscale in math and in 98th percentile in English?
There IS no general experience. Really. How I feel about, say, the AP Macroeconomics exam is utterly different from how I feel about the PISA is utterly different from how I feel about the old SAT and I don’t have an opinion on the new one yet.
How familiar are you, in your research, with all these different tests? How could anyone with exposure have the same general feel about them?
They gave up pretending to measure “innate ability” a long time ago. Decades, really. This is the current description of the new SAT:
Like I said, for my purposes, it does not matter how you choose to respond to the survey. So long as you’re thinking seriously about standardized testing and being genuine in your responses, it’s not going to make a difference. If you want to answer the survey for an Iowa Basics exam you took twenty years ago, that’s fine. If you want to generalize across multiple tests, that’s fine, too.
In the rare instance where you feel that your attitude is being informed by two competing experiences (by one great test and by one substandard test, for example), and you are finding it difficult to answer the survey consistently, I recommend thinking of a specific test.
Took the survey without reading the responses in the thread.
IMHO standardized tests are a useful but limited tool and the limitations of it as a tool are not well appreciated by those who use the tool.
I think standardized benchmarks are important, somewhere above “nice” and below “necessary,” but when the act of measuring unduly impacts that which is being measured we may have a problem … and their worth is often oversold and of excessive impact. Note for example that Finland tests relatively little and puts little weight on the results yet by multiple measures, including international benchmark testing and the numbers of scientific and engineering researchers produced per capita they excel.
Disclaimer: I score standardized test open response and essay items part time, usually during spring and summer. It’s not my full time job.
I responded to the survey and I’m a strong advocate for standardized testing. I answered the question about cheating from the viewpoint that I don’t believe there are many students cheating. There is a problem with athletes using ringers to take their ACT or SAT. There is also problem with teacher and administrators trying to game the results in order to show their schools in a more favorable light.
I dislike standardized testing for the same reasons that everyone else does: teaching to the test, focusing on certain subjects and topics and ignoring others, consuming huge amounts of time and money, and producing results that may or may not reflect students’ actual abilities. I also agree that there needs to be some fair way to evaluate public schools and teachers. Do I have a solution that would solve all the problems? No.
The big problem is that you do not explain what you mean by standardised testing. The same words mean different things to different people, as lawyers continually demonstrate.
The second problem is that you have an international audience but phrase the questions as if to an American audience.
Took it but age may be a factor with me. The standardized tests I grew up with, and recall the most as examples of such tests, were the basic IQ/SAT tests. Now, judging from the experience of the last couple years, the more psychological tests one is given as part of say a job application process are more common and thought of as more the norm. The former I think may have some validity; the latter I question. You may want to clarify which form of test you are most curious about as some questions indicate one form and other questions the other.
When I took the survey, the first question I had was, “What does the survey mean by ‘standardized test’?” Would something like the Mathematics Association of America’s AIME (which is “sort of” multiple choice - every question’s answer is an integer from 0 to 999) or USAMO (where answers have to be written out, and, in fact, usually the answer itself is a proof of something) be considered standardized?
I keep hearing the term “teaching to the test” (originally from my mother, when listening to her explain why our local school district wasn’t the best-testing one in the area - “other schools teach to the test,” or at least the ones that didn’t outright cheat supposedly did). I am a believer in, “If the test teaches the right things, then isn’t ‘teaching to the test’ the object of the class?”
It comes down to four words: “What should students know?” I have noticed a trend towards “teach them how to learn things” as opposed to “facts that can be looked up,” but without a core set of facts, you could easily end up with students who believe (a) the USSR and Germany were on the same side throughout World War II, and (b) the moon landings were “recorded in a TV studio.” One disturbing Facebook meme: “My teacher told me that I needed to learn mathematics as I wouldn’t have a calculator with me at all times {insert photo of a cellphone with a calculator app} You were saying?”
