I really don’t think so. I really, truly, honestly, and sincerely don’t think so. Because:
a) If there is ever a single false claim, then there is grounds for skepticism; and
b) If there is ever a single true claim, then there is grounds for bothering to examine the facts.
Because I know for a fact that there are true claims, then my job is to determine whether this claim in front of me is a true claim. Knowing how many false claims there are does not help me make this determination.
If you disagree–if you believe that knowing the number of false claims helps me determine whether this claim is false–can you explain, specifically, how it helps?
Including every conceivable variable in a model yields diminishing returns. What is important is controlling for critical variables. Failure to do so yields omitted variable bias, which can render the effects you claim to observe completely worthless. A so-called statistic that is generated by a misspecified model is worse than worthless: it is misleading.
I did appreciate your efforts to redirect the statistics arguments in that direction. You did try. But it wasn’t our screaming at Huerta88* that surrounded your post.
That was followed by a bantam rooster who made assumptions about my position on the situation at Duke and who used diminishing and patronizing terms without any reason except provocation. I had not attacked him or his position specifically:
This post by Huerta88 was followed by another by Huerta, one by Lochdale, another by Huerta88, another by Ellis Dee, another by Huerta88, and one by DragonAsh before Monstro finally posted.
And the beautiful irony is that in the middle of this barrage of ranting and screaming, Ellis Dee, you posted this:
I agree with Ellis Dee except about who had been doing most of the screaming. It practically became his blog for a while.
I’m glad you agree that the statistics are meaningless to the Duke case.
No shock about it. The mathematical formula you used looked a bit round-the-houses to me compared to the simple method of multiplying .95 by 980, .95 by 20, etc, etc, but it’s not exactly scary that the two methods should come up with the same answer.
Oh, I already know why people take screening tests. It’s because the value of a true positive far outweighs the negative value of a false positive. If I falsely test positive for prostate cancer, I’m in for a few anxious moments. But if I truly test positive for prostate cancer, I can get timely medical intervention that may add years to my life. I don’t mind an unreliable test in that case, as long as it’s biassed against false negatives. The case is altered however when we’re talking about a testimony in a court of law. We’re not presuming that a false positive is any more acceptable than a false negative - at least, I sure hope we’re not. And in that case it is very much to the point that a 95% reliability achieved under unrealistic conditions maps to barely more than 28% reliability in the actual population.
For the purposes of the mathematical problem we were assuming so, and this was pretty much stated.
And you believe you are educating me in what way…?
All true so far as it goes, and all irrelevant to the meat of the matter: that given a mathematical problem that specifies that there is a particular pool of cars that are deemed equally likely to be suspects, and a witness who has tested on a balanced sample of red and maroon cars to be 95% able to identify red as red and maroon as maroon, the witness’s probability of correctly identifying a maroon car from a highly skewed population falls to about 28-29%.
Thank you, I already know what “random” means.
Too right. It’s proving to be about your attachment to the 95% reliability of the key witness as being applicable regardless of what the sums say.
No, you chose loftily to inform me what “random” means, but I do already know perfectly well, thanks.
Forgive my obtuseness, but I don’t see how these two are analogous at all. If it were established that the car really was maroon, and we were arguing about the unlikeliness of it being so, then you’d have a case. But in the context of this very simple mathematical problem, the colour of the car is the very question that hangs in the balance.
A better analogy would be if you told me that you had seen today’s winning lottery numbers only yesterday as a telephone number on a billboard. I might grant you an excellent memory for 12-digit numbers, but I’d still consider it more likely that you’d misremembered a digit than not. Whereas if we saw a story in the paper about an unfortunate gambler who would have won that week if only he had used his own telephone number… well, that’s a non-story: there are millions of telephone subscribers in the country, and the paper can report on any of them that happens to write in and report the so-called “coincidence”.
But in this instance, that superficial characteristic was the sole key fact that the eyewitness was able to bring.
Quite. Just as a so-called witness reliability that was established by an unrealistic test is worthless and misleading.
I suppose it helps when you are determining where to allocate finite resources. Given infinite manpower and money I’d investigate all claims as exhaustively as humanly possible. Without that, I’d prefer to concentrate on the claims that had a history of being true fifty times for every time they were false, rather than the claims for which the reverse was true. It’s a case of furthering the ends of justice to the greatest extent possible within the constraints. I don’t say I’d want to ignore the unlikely claims entirely, but I’d be inclined to a little world-weary cynicism and perhaps a touch of perfunctoriness. Of course, political concerns might oblige me to treat all claims the same, but on the whole, more crimes would go unpunished as a result.
Yeah, but also keep in mind that prostate screening tests are not given to the whole population, but rather to at risk groups. If a prostate screening test has the same parameters as your maroon-identifying witness, it would have a horrible PPV if every male in the population took the test. But when you only target men in the at-risk age group, the PPV signficantly increases. Which is why it has value.
So I just talked about prostate screening tests and why they have value. You should know by now such screening is not recommended to everyone in the population, and that’s specifically because that would increase the false positive rate.
Apply the same reasoning to this car hypothetical. A hit-and-run accident will produce a handful of suspects: basically all the cars that were observed to be in the immediate vicinity when the crime took place. This is, in effect, our “at-risk” group. It’s not the entire population of the city, state, country, world, or universe. It’s clear that the lines you have drawn for your population are completely arbitrary. You might as well be including people in Siberia as part of your “population”.
This is the part that you keep tripping up on, Malacandra. I really don’t know how to explain this any better, so perhaps we should agree to disagree. I feel like I’m talking to a wall.
You keep emphasizing the “mathematicalness” of this problem and I’m like, so what? This discussion is not about making some numbers add up to the right thing. It’s about knowing how to apply statistics properly. Two different concepts.
It’s clear that you are only parroting back what the “New Scientist” has told you because you keep repeating the same thing over and over again. We have no reason to believe that the “particular pool of cars that are deemed equally likely to be suspects” matches anywhere close to the composition of the whole population.
Then why did you ask a question that clearly demonstrated that you do not?
The witness says they saw a maroon car.
You doubt that witness.
Your reasoning boils down to the hypothetical fact that maroon cars are rare.
This is no different than doubting me when I say that a man named Bucky Bugeyed stole 10 bucks from me.
How many men out there are named Bucky Bugeyed? Only one out of a global population of 6.6 billion? According to The Logic of Malacandra , this means this was an improbable event and therefore unlikely. So perhaps I just heard his name wrong, eh? Maybe it’s really Bucky Beaver. There’s a lot more guys with that name and they do sound a lot alike. So Bucky Beaver it is!
You are committing this error, whether you are aware of it or not. Frankly, I don’t see why you can’t see that by now.
No, this is a crappy analogy.
Reliability is completely separate from positive predictive value (or probability). “Reliability” is a term that has a precise meaning in the scientific community and you betray ignorance when you use it inappropriately.
It doesn’t look like I’m getting through to you, so I quit. Hopefully all this explaining wasn’t done in vain and I gave you at least a little something to chew on.
There’s some relevance here, I just know it. takes out magnifying glass
Okay, and your grounds for supposing that the maroon-to-red proportion in the “at-risk” group will differ from the general proportion of maroon to red within a few-mile radius would be…? Oh, because the witness said it was maroon. Beg the question much?
I believe I am acquainted with the sensation. Reminds me of a line out of “Things My Girlfriend And I Have Argued About” about watching a shoal of Argument Fish being pursued by a Truth Shark, as every time we get around to the hypothetical witness’s hypothetical credibility you find some other reason to argue that the numbers we should consider are something other than the numbers in the question, mostly (as far as I can tell) because you feel that a witness who has tested 95% reliable in one set of circumstances must always remain so.
Indeed. One’s about making the explanation fit the data, and the other’s about making the data fit the explanation. “I’m, like, so what?” is not an appropriate attitude for a professional scientist to take over mathematics!
We have no reason so suppose that it doesn’t, either. Indeed, it’s not impossible that in fact none of the town’s 20 maroon cars ever drive by the location in question. But in the absence of other data - and the problem didn’t provide any other - you don’t have any grounds for supposing that the local distribution is any different from the town-wide one. Oh, except that the witness said it was maroon. :rolleyes:
As I do know very well what “random” means, I cannot very well have knowingly asked a question that demonstrates that I do not, so you can’t expect me to meaningfully answer such a question.
Good, so you’re with me so far. In the example, there’s nothing “hypothetical” about the rarity of maroon cars. They’re outnumbered 49-1 by red, and the two colours are occasionally mistaken for each other even by people with good colour vision. Right, now we’re getting somewhere…
[will to live=evaporating]Depends. How’d you learn his name? Yelled out by a passer-by who is now untraceable? Tests show that you occasionally identify a yell of “Bucky Bugeyed” as “Bucky Beaver” and vice versa? Bucky Beaver is a far more common name than Bucky Bugeyes? What are your grounds for having your supposed identification of Bucky Bugeyes taken seriously?[/wtl]
Yes, if it’s an established fact that a man named Bucky Bugeyed stole $10 from you then it is cretinous to burble “Wow! What are the odds of that?”. But if the question we are trying to determine is what the man’s name was…
Oy. :smack: Am I right in thinking that you would sink if dropped into a bath of mercury?
Suppose I say that David Beckham (England football captain) beat me up and stole my wallet. He is a well-known national figure and his face is regularly in the papers, and it is extremely likely that I would recognise him if I saw him. For all that, it is possible that someone who merely looks like him is the culprit. My recognition of his face is the sole relevant criterion for this particular question. Now, setting aside all question of what the hell multimillionaire Beckham would want with my wallet, and in the absence of other information concerning Beckham’s movements, what is the likelihood that I was robbed by one particular man in the whole country, rather than one of possibly thousands who happen to look a lot like him?
Can you not see that if my picture appears in the paper along with Beckham’s after he has been convicted by due process of law of robbery with violence against me, it is then that only an idiot would say “Wow! What were the odds of that?”?
Your well-thought-out refutation has completely drawn the wind from my sails. Oh, wait, no it hasn’t.
This is you not beating anyone over the head with your doctorate, then? :dubious: As you well know, I was not aiming for precise usage of a scientific term in addressing Maeglin but simply using the word in its colloquial sense. When struggling with the grade-school math and logic, resort to professional condescension, huh?
Whereupon the good Doctor checks out of the argument, declaring herself the winner.
Read my hypothetical and see if any of this gets through:
There’s a hit-and-run accident on the corner of South and Main St. With me so far?
A few seconds before the incident, a surveillance camera upstream from the accident catches four cars driving down the street. The front-most car is the victim’s car (who ended up dying from the collision). The three subsequent cars are considered suspects because of the timing of their passing in relation to the incident.
Two of these suspect cars are red. One of them is maroon. This has been verified by cross-referencing the license plates with registration data. The prevalence of maroons in this “population” of suspects is 33%.*
An eye-witness to the crime saw the accident and claims that the culpable car was maroon. Prosecution investigates, finds some other evidence (such as the surveillance video), and the driver of the maroon is arrested and taken to court.
The prosecution says the witness has been shown to accurately spot maroon cars with 95% proficiency. The witness goes on to testify that he saw a maroon car commit the crime.
You are a lawyer for the defense. You claim the witness is unreliable because internationally, maroon cars only make up 2% of the car population. Ergo, the witness can not be beleived and the defendant is innocent.
The jury bursts out in laughter. The judge bangs his gavel for order, but stops when she too is overcome with the giggles. She adjourns court for a recess after the baliff starts convulsing. You see, he has epilepsy and its triggered when he laughs too hard. The defendant, on the other hand, starts crying like a baby. Because he realizes that he wasted his money on a charlatan of a lawyer and knows he has no chance in hell of getting off now. You are confused by all these reactions, because according to the New Scientist, your argument was rock solid and beyond dispute. So you stand in the middle of the court room, scratching your head, breathing out of your mouth. A cowlick pops up on the back of your head.
The lawyer for the prosecution leans back in her seat and smiles, amused with the circus around her. She’s so glad that she had had the idea of slipping a copy of New Scientist in your mailbox a week before the trial. Very glad.
Roll credits.
*The witness now has an 85% of being correct. The “at-risk” population is the population you need to be concerned about. Not the entire population of cars in the whole world. Or even the country. Or even the state. Or even the city. Not even gasp the county!
It’s not condescending to advise you to use words properly, as they are defined and understood in the field of applied statistics (you know, the subject matter that we are discussing?). And if this innocuous information counts as “beating anyone over the head with your doctorate”, you need to stop being so insecure.
Everytime you use “reliable” as you are using it you are essentially saying to the world “I don’t know what the hell I’m talking about! Please laugh and point at me!”
Assuming that all guys who look like Beckham (including Beckham) have equal chance of being the culprit, the likelihood that you were robbed by Beckham is no different had you been robbed by his evil twin brother, the sexy janitor at the McDonald’s down the street, or Charlie the hot bus driver.
In the absence of evidence, means that no one has been identified as a suspect. Why should Beckham get any special considerations that Charlie the bus driver doesn’t? Charlie hasn’t ever stolen anything in his life. Charlie is a good, God-fearing man.
I’m not sure “finite resources” is the trigger for weighing likelihood; instead, I think that “inadequate resources” should be the trigger. If there are ten reports of rape and I only have the rsources to examine three of them fully, or nine of them perfunctorily, then it makes sense for me to engage in some sort of triage, deciding which cases are likeliest to be prosecutable, which rapists are likeliest to be repeat offenders, etc. Part of that awful calculus may involve deciding which allegations are likeliest to be true.
However, I’ve seen no evidence that this is the case in Durhum, that the DA’s office lacks the resources to investigate all rape allegations adequately. If we stipulate that they have the resources to perform an adequate initial investigation of every rape allegation, then do you agree that the argument about statistical likelihoods (assuming that the statement “some, but not all, rape allegations with a certain trait are legitimate accusations” is a true statement) is irrelevant?
Holy fucking hell you with the face, how do you not get this?. How many times can people try to get it through to you that talking about claims of facts are not the same thing as talking about actual, verifiable facts?!?
That is why the random number generator analogy fails. Because in that analogy you are assuming that you know the answer.
Try this instead:
I happen to have a random number generator sitting here (picks 6 numbers between 1 and 49 - just like the lotto). Let er rip. Okay, it picked 3,17, 22, 24, 47 and 49. Or so I claim. Knowing nothing about me, and not being able to verify this draw of numbers, how do you evaluate the chance that my claim is true? WITHOUT INTRODUCING ANY NEW EVIDENCE. No digital cameras or witnesses or anything. Get it?
You don’t understand the analogy, then. It’s flown so far over your head that you must have never seen it.
So let me walk you through it: I tell you a random number generator has spit out 5 integers from 1 to 100. Let’s say they are 56, 12, 4, 35, and 99.
You do the math and the odds of these 5 numbers coming up out of all the combinations of numbers between 1 and 100 is extremely low. Nevermind the fact that any combination of 5 numbers between 1 and 100 will have these same odds. You are so fixated on the odds that you swear and up down that this is unlikely.
So you say you don’t believe me because of this.
And I laugh at you.
Now please explain how this is differs from the Beckham analogy put forth by Malacandra. Since you’ve declared that I “don’t get it” surely you can explain where my logic has gone off track.
Of course I don’t believe you. Because the odds are so against it. Kind of like the lottery. They actually force you to verify your claim. And if you can’t, they laugh at you.
All you’ve done is change the hypothetical to place yourself in the position of “knowing the answer” again. Answer the question at the end of my previous post please. With my claim of generated numbers. And you being the one unable to verify the truth.
How do I evaluate the chance that your claim is true? I look at the benefits to you of lying to me, how likely it is that you know that I can’t verify your draw, and the potential problems caused by my incorrectly evaluating the chance that the claim is true.
If it turns out that there’s a huge problem caused by my incorrectly evaluating the chance, and there’s no probelm caused by my refusing to evaluate the chance barring further information, then I refuse to evaluate the chance.
Yes, I know it’s not the answer you’re looking for–but if you further modify the analogy, you remove it from the realm of relevancy.
I’m really trying to understand your position. Maybe I’m not reading you correctly.
Because the odds that the RNG would have generated that particular string of numbers (out of all the other combinations it could have generated), you doubt my claim?
**Even though the probability that particular string of numbers being generated is no different than any other combination? **
Is this what you are saying?
I has fuckall to do with “knowing”, and all to do with “Why shouldn’t you believe me?”
Should you be skeptical of a claim that I drew the number 4 out of a hat containing 100 numbers? The probability of this is low (1/100), right? But that factoid alone should not warrant skepticism. Especially since me drawing any number has 1 to 100 odds.
If someone claimed that the RNG returned a sequence of numbers that did not appear, to my eyes, randomly selected (i.e. it looks like a series generated through some form of intelligence) then yes, that would give me pause. Because it goes against my understanding of nature.
However, I don’t have to compute probability to reach this conclusion because the mathematical probability of getting those numbers is actually no different had the sequence been truely random. In other words, skepticism comes from my awareness of the world (common sense) and not math.
Now if your sequence doesn’t strike me as being “unrandom”, then I have no reason to doubt you. It sounds plausible because it doesn’t go against my worldview. Probability has nothing to do with it.
This does really baffle me. I imagine you pulling a number out of a hat, looking at it, saying, “I drew a 4,” and folks shouting, “Bullshit, you lying whore!”
Why?
If you want to be skeptical of ywtf’s claim, then have at it. If you want to demand additional corroborating evidence before you’re satisfied that she drew a 4, good for you!
But if you believe the hat has 99 4’s and one 17 in it, and she says she drew a 4, wouldn’t you also want corroboration, if the number she drew had some impact on the world (let’s say, if she drew a 4, it’d send three guys to prison for a long time)? Assuming that you’ve got the resources to achieve corroboration, why would you demand more corroboration in the first case than the second?
Nobody’s objecting to the calls for corroboration. Folks are objecting to saying that the call for corroboration should be affected by the rarity of the claim, when the claim happens on an annual basis.
The random number generator was my hypothetical. In my hypothetical, I said this:
So that’s a given. (I believe at the time we were talking about sixteen numbers.)
The odds are the same for any sequence of sixteen numbers between 0-99. Unbelievably slim.
And yet we know before the numbers are generated that one sequence of sixteen numbers will beat the odds.
The series of numbers will be generated on live television at the Oscars with VegemiteMoose, YWTF, LHoD, Roger Penrose, Cecil, and Tomndebb in attendance as witnesses and participants. Each of them will contribute one number. VegemiteMoose will get to choose the order and place in which people will insert numbers. The other numbers will be generated by the random number generator.
The sequence of sixteen will prevail in a parallel universe if something goes wrong in this one.