Lighting Rod Conclusion - Confirmation Bias?

Cecil's Columns/Staff Reports
1

Link to today’s column about Lightning Rods

Ok, my take on today’s article is that the efficency of lightning rods falls into the SDMB/GD’s legendary Confirmation Bias. There are:
[QUOTE=Cecil]
…no control group, no point of comparison, nothing telling us that unprotected towers of similar design in Florida were clobbered.
[/QUOTE]
So there’s no scientific evidence that not having Franklin Rods helps.
[QUOTE=Cecil]
The whole thing started when a proposed standard for a new lightning protection technology called early streamer emission (ESE) was rejected. There’s no proof it works, the experts said. But you know what? We’re not so sure there’s proof conventional lightning rods (known to the trade as Franklin rods) work either.
[/QUOTE]
And no scientific evidence that they do work. Per Cecil’s column there’s only:
[QUOTE=Cecil]
A lot of… data [that] is old and anecdotal and was casually collected by people without scientific training; in too many cases it boils down to “We used to have a lot of lightning damage, but after we put up Franklin rods it stopped.”
[/QUOTE]
It sounds to me like there was a problem, someone (Ol’ Unca Benny, the C-note guy) did something to solve the problem and it worked, whether it was supposed to or not. Scientific method be damned, Cecil’s “gut feeling” along with every body elses "gut feeling"s say it works so it works.

Never mind the fact that they do work.

My point is this: Is this not confirmation bias? Is this the commonly accepted, government approved stuff of snake oil and faith healing? At what point does an absolute rejection of confirmation bias give way to common sense? At what point do we look at something (anything) and say “Damn it, it works. We don’t know how or why but it works.”? At what point do we sacrifice intellectual pride for the common denominator regardless of study or studiability? At what point do we stop making up words like studiability?

Just askin’ is all.

2

No idea what you’re trying to say.

It looks like you’re proposing we just accept a history of anecdotal evidence and not bother to do a controlled study. Is that right?

3

Well, that’s what I’m saying, Baldwin. At what point can we recognize confimation bias? When there is contrary evidence that is being ignored. There may be little good, solid, scientific-with-lots-of-charts-and-computer-models evidence that Franklin rods work, but I don’t see any research or even anecdotal evidence that says that they don’t. Not even the ESE people disagree that Franklin rods work pretty well. So, at what point can the government say, “C’mon, guys, we know they work and you know they work. We may not know exactly how, but this one is obvious so can we save some of the taxpayers’ money for once?”

Good ol’ anecdotal evidence: I happened to be looking at our rod’s grounding wire twice when our house was hit (tall house, taller TV antenna, where the rod was placed, at the edge of a large field–we got hit a lot). Both times the cable was surrounded by glowing plasma as the charge bled off to ground. And none of the times we were hit did the house catch fire or suffer structural damage. It never wrecked the TV, either, though that might be due to old-fashioned tube technology and a modern TV would be toasted.

4

This is not an example of confirmation bias. In confirmation bias, there is a rejection of non-conforming data and acceptance of conforming data. Here, there simply is a lack of non-conforming data.

5

[QUOTE=dropzone]
There may be little good, solid, scientific-with-lots-of-charts-and-computer-models evidence that Franklin rods work, but I don’t see any research or even anecdotal evidence that says that they don’t.
[/QUOTE]
I don’t have the background to comment specifically on lightning rods, but “prove they don’t work” is not relevant to the scientific method.

If a claim is made that an effect is produced by some form of technology, it’s up to the claimant to furnish affirmative supportive evidence. Others may have the time and money to conduct studies showing otherwise, but it’s not incumbent on them to do so. In the absence of positive evidence, the claim is unproven.

In Cecil’s words “Two hundred fifty years of observation counts for something”.

It does, but less than you’d think. There are many examples of seemingly compelling anecdotal evidence that get overturned by careful study (example - traditional forms of medicine that are followed for hundreds and thousands of years with lots of testimonial support, but ultimately prove to be ineffective or harmful when studied in a controlled situation).

6

[QUOTE=Leo Tolstoy]
“The most difficult subjects can be explained to the most slow-witted man if he has not formed any idea of them already; but the simplest thing cannot be made clear to the most intelligent man if he is firmly persuaded that he knows already, without a shadow of doubt, what is laid before him.”
[/QUOTE]
(from DSYoungEsq’s link)

This may well be where I am confused (and, unless I’m more messed up in the head than I think I am, many others may be confused as well). I have always held Confirmation Bias as a biased way of thinking one way or another, regardless of evidence, based on whether or not the expectations of a phenomenon were achieved (e.g. I pray for $100, a $100 check I wasn’t expecting arrives in the mail, badda-bing the prayer worked) even when those expectations are eccentric or counter-intuitive (e.g. I’m hungry… I’m broke… I pray for food… a $100 check I wasn’t expecting arrives in the mail… profit!). I had not considered the rejection of contrary evidence where such evidence exists.

So what sort of bias is it when there is no viable evidence supporting a position (repeatable scientific stuff, not just anecdotes) but no viable evidence contrary to a position? At what point does an unproven hypothesis become usefull (personally, financially, scientifically, whatever) when a controlled observation cannot be made and only anecdotal evidence exists?

7

[QUOTE=Jackmannii]
I don’t have the background to comment specifically on lightning rods, but “prove they don’t work” is not relevant to the scientific method.
[/QUOTE]
Would a study that showed no difference between the lightning damage to buildings with or without lightning rods not, in layman’s terms, “prove they don’t work” in the same manner as one that showed no difference in patients taking powdered cattail root and those taking a placebo? I am not talking about the goal of a study but its results, and there appear to be no such studies of lighning rods. As crappily acquired and organized by the standards of good scientific method as the evidence may be, it remains that damage to protected buildings due to lightning appears to be less than expected and has so for 250 years. I would not stand in the way of any studies that attempted to explain why lightning rods appear to work, or even if they really do. I merely suggest that there is some difference between the apparent fact that lightning rods protect buildings and claims that copper bracelets cure arthritis.

8

Confirmation bias or not, Cecil has a typo in the report. All the way through he refers to NFPA 780 as the standard covering lightning rods, until the last paragraph where it’s NFPA 708. A trip to the NFPA website confirms that 780 is correct.

A neighbor’s house was struck by lightning where I lived some years ago and, while it did not start a fire, damage was quite noticeable. Several bricks were knocked off of the chimney – the presumed strike point – and the plaster board up by the ceiling right by the fireplace was blown off. Worst of all was that they had radiant heating wires in the ceiling, and that got fried. You could see the plaster discolored by the wire’s tracings back and forth and the thermostat was slagged. I don’t know what they did to repair it; we moved away shortly afterward.

9

Radiant heating wires in the ceiling? Y’all do things a little different in Arizona don’t ya. We put these in the floor.

10

[QUOTE=nd_n8]
(from DSYoungEsq’s link)

This may well be where I am confused (and, unless I’m more messed up in the head than I think I am, many others may be confused as well). I have always held Confirmation Bias as a biased way of thinking one way or another, regardless of evidence, based on whether or not the expectations of a phenomenon were achieved (e.g. I pray for $100, a $100 check I wasn’t expecting arrives in the mail, badda-bing the prayer worked) even when those expectations are eccentric or counter-intuitive (e.g. I’m hungry… I’m broke… I pray for food… a $100 check I wasn’t expecting arrives in the mail… profit!). I had not considered the rejection of contrary evidence where such evidence exists.

So what sort of bias is it when there is no viable evidence supporting a position (repeatable scientific stuff, not just anecdotes) but no viable evidence contrary to a position? At what point does an unproven hypothesis become usefull (personally, financially, scientifically, whatever) when a controlled observation cannot be made and only anecdotal evidence exists?
[/QUOTE]

There’s plenty of contrary evidence surrounding your example. All those times when you were broke and prayed for money and no money appeared. You just forget about those later on and only remember the times that it worked. That’s the almost textbook example of confirmation bias.

11

I haven’t read the reports indicated, but there certainly is a history of testing of lightnng rods that goes beyyond looking at actual lightning damage. In Viemeister’s ** The Lightning Book** (1972) there are reports and photos of “lightning” tests using huge lightning generators (basically big Van de Graaf generators, I gather) and scale models of buildings with and without lightning protection, in various arrangements.

It’s probably the closest thing you’ll get to reproducible tests of lightning rods, using directable sources and control experiments. The results are pretty unequivocal – lightning rods DO protect property if properly used. You can see a picture of the strikes around a minature rod-protected house in the book. there aren’t any strikes in the rod-protected “zone of protection”.
There are arrangements and formulae for the placing and spacing of lightning rods, developed using these tests. You don’t install lightning rods by guess and by god and just sticking them in place. Saying that there’s no reliable basis for saying that lightning rods work strikes me as bizarre.

– CalMeacham, whose brother-in-law installs lightning rods.

12

Anecdotal/observational evidence differs in quality, based on who is doing the observing and what their education and biases are. I was noting that 1) it is nowhere near as strong as, for example, that to be obtained through well-conducted comparison trials, and 2) “Prove I am wrong” is not accepted in lieu of evidence in the scientific method. Every half-baked weasel* with a goofy theory rolls this one out to defend his lack of data and to try to get sympathy from people willing to believe that the nutty theory would be supported if only there wasn’t a Gummint/Big Pharma/Illuminati conspiracy to suppress research into it.**

[QUOTE=CalMeacham]
CalMeacham, whose brother-in-law installs lightning rods.
[/QUOTE]
Aha, a Conflict Of Interest (betcha you get a good deal on them things. Not everyone is lucky enough to have a brother-in-law in the lightning rod business)!
*This description was not meant to refer to anyone posting in this thread.

**As Groucho Marx once said, “Who are you going to believe, me or your own eyes?”

13

[QUOTE=Exapno Mapcase]
There’s plenty of contrary evidence surrounding your example. All those times when you were broke and prayed for money and no money appeared. You just forget about those later on and only remember the times that it worked. That’s the almost textbook example of confirmation bias.
[/QUOTE]

This doesn’t necessarily mean the evidence is contrary (it doesn’t necessarily mean it’s not either) it simply means that while the expectations were indeed met they may not have been the expectations that were specifically desired.

It is apologetic to say that when expectations are met in an unexpected manner that The Lord/TFSM/Sid Ceaser works in mysterious ways, and besides this is neither my point nor the forum for it.

I realize that scientific research needs be conducted without a goal in mind to unbias the research, but expectations (specific or general) must exist for the research in the first place. The difference lies in the difference between expectations of a desired result and the goal of a desired result. Does failing to meet an expectation but attaining the desired result falsify the hypothesis or simply change the expectation? Can the two be seperated without bias?

[QUOTE=Jackmannii]
Every half-baked weasel* with a goofy theory rolls this one out to defend his lack of data and to try to get sympathy from people willing to believe that the nutty theory would be supported if only there wasn’t a Gummint/Big Pharma/Illuminati conspiracy to suppress research into it.**
[/QUOTE]
Just because you’re paranoid doesn’t mean they’re not out to get you.

14

[QUOTE=Jackmannii]
2) “Prove I am wrong” is not accepted in lieu of evidence in the scientific method. Every half-baked weasel* with a goofy theory rolls this one out to defend his lack of data and to try to get sympathy from people willing to believe that the nutty theory would be supported if only there wasn’t a Gummint/Big Pharma/Illuminati conspiracy to suppress research into it.**
[/QUOTE]
Oh, I see. Your heart was broken by a half-baked weasel with a goofy theory so now you see any statement like “I don’t see any research or even anecdotal evidence that says that they don’t (work)” as equal to “Prove I am wrong.” (patting your hand condescendingly) I understand and I hope you can put your pain behind you and learn to love again. :smiley:

15

[QUOTE=dropzone]
Oh, I see. Your heart was broken by a half-baked weasel
[/QUOTE]
No, I’m just tired of them nipping at my ankles. :slight_smile:

16

That’s what you get for undercooking them.

17

That’s a fascinating column. I had no idea the efficacy of lightning rods has ever seriously been in doubt since Franklin’s time (and I grew up with Robert Lawson’s charming kids’ book Amos and Me, about the mouse who gives ol’ Ben all his best ideas, including the lightning rod).

18

The mouse’s name was Amos; the book, being written in Amos’s voice, is Ben and Me. Disney made a featurette of it.

19

[QUOTE=nd_n8]
This doesn’t necessarily mean the evidence is contrary (it doesn’t necessarily mean it’s not either) it simply means that while the expectations were indeed met they may not have been the expectations that were specifically desired.

It is apologetic to say that when expectations are met in an unexpected manner that The Lord/TFSM/Sid Ceaser works in mysterious ways, and besides this is neither my point nor the forum for it.

I realize that scientific research needs be conducted without a goal in mind to unbias the research, but expectations (specific or general) must exist for the research in the first place. The difference lies in the difference between expectations of a desired result and the goal of a desired result. Does failing to meet an expectation but attaining the desired result falsify the hypothesis or simply change the expectation? Can the two be seperated without bias?

Just because you’re paranoid doesn’t mean they’re not out to get you.
[/QUOTE]

There are two major types of scientific research.

Fundamental research is undertaken just to find out the basics of how things work. There are no stated or unstated expectations, because no knowledge of the results is present before the experiments begin.

Hypothesis testing, OTOH, is done when a theory makes predictions about the outcome of an event but the event has not been formally examined. The stated goal of the experiment is to prove or disprove the stated hypothesis. It must always be explicit and stated in exact, formal terms or else the experiment is scientifically worthless.

The goal of hypothesis testing is to take into account every single trial. The scientist must look at all events, including the ones where no results occur as well as the ones in which contrary results occur.

To use a real world example, that’s where all psychic research fails historically. (Well, aside from the tests that were fraudulent.) When outsiders go over the history of tests made by Rhine and other researchers they always find a pattern in which tests without positive results are explained away somehow. The testee wasn’t ready, or the conditions weren’t right, or they’re just somehow ignored. Only the positive results count.

For most everyday observations the same pattern exists. Positive results are remembered. Negative results are forgotten or ignored or explained away. There are no control groups, no null hypotheses, no accurate statistics. That’s why scientific examinations have disproven a zillion popular beliefs, from astrology to the special chaos that happens only at a full moon. The reverse is how science changes even major theories that everyone had previously agreed upon, when results come up that can’t be accounted for by the older theories. The theory has to explain all the results or else it’s not a complete theory.

And whether you’re paranoid or not, all conspiracy theories are wrong. Period.

20

[QUOTE=Exapno Mapcase]
And whether you’re paranoid or not, all conspiracy theories are wrong. Period.
[/QUOTE]
Yeah, conspiracy theories like “there is a large group of people, mostly Italians, conspiring to commit crimes including, but not limited to, narcotics distribution, gambling, and prostitution,” or “there was a group of people who conspired to assasinate President Lincoln.” :rolleyes: