College Board runs the PSAT/SAT and ETS runs the GRE, Praxis, et al.
Suppose I am taking one of those tests and I do the whole security gamut. No cell phone. All personal items turned in. Turn out pockets. Etc. And this scenario could be either their proctored testing in a group, at home or at a computer testing center.
Now suppose I do something remarkable like take the entire 2 hour test in 15 minutes with 90% accuracy. College Board or ETS is sure I cheated because in their opinion no one can take a test that quickly with that many right answers. Can they, under their and conditions, just say, “We’re pretty sure you cheated. We have no evidence of cheating. We can’t say how you cheated. There is nothing suspicious besides your score and elapsed time. But regardless we are invalidating your score based on our suspicion.”?
To take the GRE as an example, see ETS’ policies here (pdf from the ETS website). It says (on the page numbered 19; 21 in the count of the pdf file):
ETS may also cancel scores if, in its judgment, there is substantial evidence that they are invalid for any other reason. Substantial evidence means evidence that is sufficient to persuade a reasonable person; the substantial evidence standard is lower (i.e., requires less proof) than the reasonable doubt, clear and convincing, and preponderance of the evidence standards. Evidence of invalid scores may include, without limitation, discrepant handwriting, unusual answer patterns and inconsistent performance on different parts of the test. Before canceling scores pursuant to this paragraph, ETS notifies the test taker in writing about its concerns, gives the test taker an opportunity to submit information that addresses ETS’s concerns, considers any such information submitted and offers the test taker a choice of options. The options may include voluntary score cancellation, a free retest, a voucher for a future test or arbitration in accordance with ETS’s standard Arbitration Agreement.
So ETS is quite generous to itself in lowering the standard of proof to which they need to have evidence that you may have cheated; it’s lower than the “reasonable doubt” standards that would apply in criminal law and even lower than the “preponderance of evidence” standard that applies in civil proceedings (where they would need to show that cheating is more likely than the absence of cheating). But it’s not complete arbitrariness (what used to be called “in its unlimnited discretion” or something like that in old legal documents) either; a clause to that effect would probably be unenforceable in many jurisdictions. From the examples they give (unusual answer patterns, inconsistent performance) I would say that simply being “too good” is not enough if they don’t have more factual evidence than that.
They can and wil investigate and potentially invalidate if there is a super-dramatic improvement, like ypu go from the 26th percentile to the 75th percentile in two consecutive tests, not too far apart.
They are pretty opaque about thier methods.
So would the board have a back channel to tell prospective colleges “watch out for this guy”? Sort of like an asterisk beside the statistic, like in sports?
I would think that answering the test correctly in far too little time would qualify as inconsistent performance - inconsistent with reasonable real-world performance. It’s the marginal cases that would be harder to call. Finishing a bit early with a very high score that does not match your academic record, for example.
I don’t think that’s what they mean by “inconsistent”; I would suppose that this wording means “inconsistent within the overall test”. Things like doing inexplicably better on some questions than on others. But it may well be that ETS is intentionally vague in this wording so as to keep options open.
On the old-style pencil-and-paper, fill-in-the-bubble standardized tests that I’m familiar with, they did not keep track of how long you took to finish; they just gave you a set time limit in which to do as well as you could, with no bonus or even any way of recording if you finished early.
I’m not familiar with the computerized versions that are common nowadays, but I suspect they can keep track of how long you take. And if so, they would have records of how long all of the many, many people who have done the test have taken. So they wouldn’t have to say anything as vague as “In our opinion, no one could do this well”; they could say “No one has ever done this well” (if that is, in fact, the case).
I wonder if that is sueable, beyond the “you can sue anyone for anything” mantra. If you paid for the test then they owe you the scores unless they can prove to a reasonable third person you cheated.
Sounds like they let you retake the test for free.
If you really did that well, then you can do so again. If you repeat your performance, then they could say, “Congratulations, no one has ever done this well on the test, here’s a full scholarship to anywhere you want to go.”
If so, I wonder if they change the supervision or would it be, “You got another perfect score. Guess you cheated again. Let’s invalidate the score a second time.”
And if I have to give up my time and brain energy a 2nd time and show I wasn’t cheating, I feel they owe me something - even if only a really good high five.
I would assume that when you sign up to take a test, you agree to abide by their policies, as spelled out by wording like what @Schnitte quoted.
I did a little searching and found one example of someone who attempted to sue over having her SAT scores flagged (but who later dropped the suit, according to the follow-up article):
I’d assume that since the whole point is to make sure that you made the score on your own, they’d ensure the proctor is qualified to monitor you properly.
If you did better on the test than anyone else has ever done, then taking it again shouldn’t be too much of a struggle. Maybe you don’t do quite as well, and instead just do better than 99.9% of anyone else has ever done.
And if you did that well, then you’d likely get a full ride scholarship to anywhere you wanted to go.
But it is the principal of the thing. They can invalidate a score and not have to prove the test-taker cheated. It is up to the test-taker to prove (to a nebulous and unknown standard) that they didn’t cheat. That system is wrong.
I guess that means that they just need to do a strip search and internal cavity search before they give out the test the first time.
Some things to keep in mind:
- They have a really strong incentive to guarantee that their tests are free from cheating. If not, the colleges, scholarships, etc. that look at test scores could easily dismiss them. “This C student got a high score on the SAT? Must have cheated somehow.”
- It behooves them to administer the test in such a way as to make cheating, especially undetectable cheating, as difficult as possible.
- They have a lot of knowledge and experience—way more than you or I—about how test-takers cheat or try to cheat, how to detect this, and what honest vs. dishonest test results look like.
Sure, but they also have a strong incentive not to annul good scores arbitrarily. If they did, the value of the scores for the selection boards of colleges, scholarships etc. would equally suffer, and students would hesitate to take the test. There’s a fine line here that they need to walk to avoid both being too lax and too strict.
RIght, but that’s kind of a straw man since I haven’t seen any evidence that they ever do annul good scores arbitrarily.
From the followup article I linked to above:
(bolding mine)
My point is that it’s too simplistic to say that ETS would have an incentive to be super strict because it’s preumably better to annul an innocent test acer’s score than let a cheater get away with it (“err on the side of caution”).
The key is they said the girl’s answers correlated too closely with those of other(s) taking the test. Presumably the implication is that they suspect the person moderating the test room was allowing the group to discuss answers, or at best was oblivious to someone “assisting” her with her answers. Test validity is only as good as the test supervisor(s) reliability. At least an online test removes the possibility of someone else filling in two answer sheets at once.
I suppose too, it’s good for the reputation of the test for it to be known that they will challenge suspect results on occasion.
I’m assuming it’s only a matter of time before a video record of the test room becomes a standard part of the test even for group tests with a supervisor.
They don’t do it often. It is exceedingly rare, in fact, and I assume only when they are pretty damn sure. I’ve worked with literally thousands of students on this and never seen it, personally, although I have seen screenshots of the letters some people have gotten on reddit.
They have an incenitive to keep the test reliable, but they also have an incentive not to get a reputation for arbitrarily cancelling scores.
They aren’t really that worried, imho, about one-off cheating that can’t be scaled up. They don’t like it, and they do their due dilligence (truly), but you can’t stop the kid with a similar looking older sibling. The real issues are when actual keys get stolen and widely distributed, or when testing centers are “in” on it. Thats why weird patterns of wrong answers and the like really get their attention.
I took it as she was looking at the answers of the people next to her and copying, not that she was assisted by them.