That all sounds nice, but in essence you’re not saying anything greater than, “I have confirmed that the output of this guy’s programs/math is indeed what he says it is.”
Why do you think that making source code public aids reproducibility? The point of reproducibility is to see if you can accomplish the same thing as someone else, using the same methods, ya? The method is not the same thing as the implementation. Confirming the implementation comes to the output that the person who made it says it did is doing nothing more than making sure they didn’t make up numbers wholecloth. Ultimately, that’s rather a useless pursuit since science would be better served by trying to replicate his method. Then if you get the same result–free from the original persons implementation–it adds weight to the theory.
Which is again all separate from critiquing a person’s method. If you think their method is flawed, then you go and do your own research and get it published in Nature and let the rest of the experts see if they find your argument for your methodology is more convincing as the best method than the one you are trying to oust.
This is the scientific method, regardless of what Whoever might like to convince you otherwise, and it makes significantly more sense.
I don’t know how you reach these conclusions. I don’t think his opponents would say there is nothing wrong with his work. And, the NAS report reached very different conclusions than he did…In fact, you have labeled it as “schizophrenic”, which seems to serve as an excuse to then ignore 99% of it, including most of the executive summary, and instead grab one sentence from one page and one from another and cobble it together to reach conclusions very different from the NAS conclusions.
As for the Wegman panel, that was a panel picked by the Republican majority on the Congressional committee to reach the conclusions that they wanted it to reach. Besides which, it studied only a very narrow question.
By the way, it is possible to do mathematics correctly but still reach totally insane conclusions. After all, I haven’t found an explicit mathematical calculation that is in error in Gerlich and Tscheuschner, but it doesn’t mean their paper is not complete and utter crap (as I think even you have pretty much admitted). McKitrick’s paper on the existence of a global temperature suffers from a similar problem. One could probably even say this about Douglass et al…I.e., they correctly calculated the standard error for the models; it was the wrong thing to use to compare the models to the observational error but they didn’t make a mathematical error in their calculation.
jshore, I challenge you to find a single statement of McIntyres that was found to be incorrect in either the NAS report or the Wegman report. You appear to not know what it was that McIntyre said, if you think the executive summary disagrees with McIntyre’s statements.
So, I await your list. This would be a list of McIntyre’s statements, along with a reference to where in the NAS or Wegman reports they say each one of them is wrong. I’m sure it won’t take long to make a list, as you wouldn’t make such sweeping statements without evidence in hand.
w.
PS - I notice that your incessant hominems now extend to Wegman. I mean, he must be suspect, despite being one of the pre-eminent US statistictians, because he was “picked by Republicans to reach the conclusions they wanted … to reach”. I doubt very much if you’d have the balls to accuse Wegman to his face of having his opinions be for sale in that manner, but you are willing to libel him here …
But if you are so sure that his conclusions are politically motivated, I’m sure you’ll be more than willing to point out the flaws in his analysis … once again, I’m willing to wait, because I’m sure you wouldn’t accuse a scientist of providing an opinion for political reasons without evidence …
My question is, why is it you are so willing to accuse me, and Wegman, and McIntyre, of playing fast and loose with the truth, but when it comes to Mann, and Wahl, and Amman, and Hegerl, and Thompson, suddenly you are struck dumb. What’s up with that?
But I await your list of Wegman’s errors regardless.
PPS - You say
On my planet, “doing mathematics correctly” includes using the appropriate procedures for the problem.
On yours, it doesn’t, so you can reach “insane conclusions”.
To set your mind at rest on this question, tomndebb, I just want to tell you that you are right, and that I feel very bad about apologizing to MrDibble for having agreed with him. As part of my commitment to making myself a better person in your eyes, I promise I will never again apologize to MrDibble for agreeing with him.
We now return you to your regularly scheduled programming.
jshore, gotta run, but thanks for your comments on the Koutsoyiannis paper, I’ll come back to it. Did you see Koutsoyiannis’s reply to Gavins comments?
Perhaps that might be all that you would confirm. Me, I confirm and investigate a host of other things - the data itself, the type of analysis used, whether his choice of mathematical assumptions (I.I.D. vs LTP, for example) matches the problem, whether he has considered other possible explanations for the phenomenon under study. In addition, I typically use the opportunity afforded by having the code and the data to do things like “what-if” analyses, to see if sub-sampling of the data gives more information, and the like. It gives me the opportunity to use the same method on other datasets. It lets me see what is implied by the choice of which proxies he used, and which proxies he didn’t use … and the list goes on and on, that is by no means exclusive.
In short, if all you would do if when given the data and code is confirm the results match, then you don’t understand the process, and I’d never hire you as my auditor …
Making source code public aids reproducibility in a couple of ways. First, it allow other researchers to determine whether there are any fundamental flaws in the analysis (e.g. as in the case of the Hockeystick) . I don’t know about you, but before going out to reproduce something, I prefer to know that there is a real phenomenon to reproduce, not just someone’s foolish mathematical mistakes. Once that is done, then the real work of reproducibility can begin.
Next, having the source code available lets me see what the researcher actually did do, rather than what he says he did in the manuscript. I do not mean that he is trying to conceal or distort, or that his work contains errors. I only mean that the description of a mathematical process in a scientific paper is often incomplete, partial, shortened for reasons of space, or unintentionally misleading.
And clearly, knowing exactly what a researcher did do, and exactly how he did it, is helpful in trying to duplicate either the method or the execution, ya?
w.
PS - You say “Confirming the implementation comes to the output that the person who made it says it did is doing nothing more than making sure they didn’t make up numbers wholecloth. Ultimately, that’s rather a useless pursuit since science would be better served by trying to replicate his method.”
Since “make up the numbers wholecloth” is exactly what Mann did with some of the Hockeystick proxy data, exactly why is confirming the implementation “useless”? In addition, confirming the implementation allowed us to show that Mann lied through his teeth when he said he didn’t calculate the R^2 statistic for the 15th century results.
So I can understand why you might want to argue against this level of confirmation, and I don’t like it myself, but unfortunately, given the current state of climate science, it has proven to be necessary.
Unfortunately, they also didn’t read the Wegman report, which says:
The main false claim in the RC postings is the idea that the problem was just the improper centering of the data. All that did was make the procedure mine for Hockeysticks.
As RC points out, you can get to the Hockeystick without the improper centering of the data. What they curiously neglect to say is that without the bristlecone pines, there is no hockeystick. Which means that the whole edifice rests on the pines, which have been shown by a variety of authorities to be worse than useless for temperature reconstructions. Wegman says:
So no, RC did not find any errors in the M&M Hockeystick analysis. In fact, the best statistical authority to examine Mann’s work says “it is not clear that the hockey-stick shape is even a temperature signal …”
Instead, RC did what they always do - post ridiculous assertions, and then censor anyone who disagrees, to make the consensus visible … anyone who depends on RC for statistical authority is either a fool or a true believer.
wevets, sorry I missed this earlier, it’s been a little chaotic. Your post is good news, that you are ready to talk intelligently about the issues.
So … what is your take on how Linah Abaneh’s PhD Thesis affects Graybill’s Sheep Mountain proxy data, which is at the core of the Hockeystick?
w.
PS - for those still interested in whether errors have been found in the M&M analysis of the Hockeystick, see here.
And for those looking for a layman’s view of the Amman/Wahl controvery (they claimed to have validated the Hockeystick), there is a very well written article here.
Finally, for those interested in what real statisticians (as opposed to those ever-censorious folks at RC) has to say about M&M and the Hockeystick, see the Wegman Report and the Response to Stupak (pdfs). Wegman talks at length about the erroneous claims made at RealClimate and cited by MrDibble above, such as the risible claim by RC that the Hockeystick method of centering is correct … about which the Wegman report says (emphasis mine):
A couple of notes on this:
Despite the claims of Sage Rat that computer code is immaterial to replication, Wegman et all clearly find that the lack of data and code stops them from reproducing the work of Mann et al.
RC says that Mann’s method (improper centering) is 100% correct. Wegman makes no bones about it, he calls Mann’s method an “error”. Describing his qualifications, Wegman says:
So you can either believe Wegman and his panel of expert statisticians, or you can believe Mann’s acolytes at RealClimate, all of whom failed to notice Mann’s error when it occurred and who now want to tell you that it was perfectly fine …
intention, this is an offiocial Warning that you are to stop hijacking this thread, dancing around trying to be clever in some sort of sophomoric game (i.e., being a jerk).
I really do not know what your problem is, but turning a simple request to avoid insulting people into a multi-post incipient meltdown is not the way to behave in this Forum.
You are to make no more snide comments in this thread (or this Forum) on the matter of MrDibble or my Moderating. Take it to the appropriate Forum or don’t post it.
Which is what peer reviews do, except by people who are better aware than you of what all other explanations there might be and what all could have been checked, and more importantly whether any of it is likely to have a sufficiently significant bearing on the result that it should be included.
Ultimately, there’s an infinite number of things you can include to make any test more finite. Getting caught up in trying to solve for them all is a sucker’s trap. You only need to solve for the biggest ones and, should it ever become necessary to do more testing, add on a few more with subsequent testing.
Scientists aren’t auditors and science isn’t a process of mutual auditing. Regardless of what you or McKitrick might desire, the system is what the system is and complaining because they use a different method to confirm results than you do is nothing more than petulant whining. If you (e.g. McKitrick or whoever) want to add productively to the process, then you’re better off served to figure out what the process for contribution is, how it works, and how to use it effectively.
No. A scientific report isn’t accepted for publication unless the methodology explained in the paper is sufficiently explained to be replicated.
The implementation itself is assumed to be flawed by everyone. Like I noted above, every implementation is going to be dealing with a subset of possible mitigating factors, and more importantly there is no end to debugging any process, which is part of the reason that auditing isn’t the scientific way–there’d be no end to it. (The other reason being, most likely, that people who want to discover new things make good scientists, but poor auditors. And people who simply want to audit are neither inclined nor capable of making discoveries.) More importantly, access to the implementation makes it more likely for people to copy and paste and repeat the same errors of implementation. When shielded from it and dealing only with the highest view, their results are guaranteed clean of such issues.
And like I’ve noted before in these threads, “In science, any one point of data or study is doubted to be accurate. Decisions aren’t made based on one study nor on one methodology. Saying that you have doubts about a single study or methodology is redundant in science, and so it’s rather a silly argument to use.” Evidence in the scientific world is created via quantity, not quality. Certainly a basic minimum of quality is necessary for the quantity to mean anything, but still the point is that no one expects any one study to be meaningful.
The Mann Hockey Stick is a great example, and is amusing for being just as much a debacle for Mann as for McKitrick and McIntyre.
No one assumed it to be necessarily perfectly accurate. But, since you could eyeball all the various graphs that were being merged and see that it would end up as a hockey stick, it didn’t strike most people as being a large issue to verify the perfect accuracy.
McKitrick and McIntyre’s output graph was equally if not an even greater hockey stick than Mann’s since it drifts downwards before turning up instead of going from flat to upturned. So by removing any possible upturn towards the end of the graph, the made the hockeystick more pronounced.
Further research (PDF) into both Mann’s and M&M’s results confirmed that any possible upturn their might have been, while unnecessary, was insignificant to the degree of being perfectly ignorable.
Mann’s graph was one of several (PDF) which were independently created and came to the same result, entirely free of any particular flaws Mann might have had in his implementation.
So overall, a lot of time was wasted on something that had no lasting value whatsoever, adding more redundancy to what is an already redundant process–and one which is more free of copying the same flaws as others might have used in their implementations than the one you suggest.
Then don’t become a scientist. That’s the sort of thing they love to do, and we should all be grateful for it.
That’s not what peer review does, it is not even what it is supposed to do. At best, all a peer review is supposed to be is a days look for obvious errors, day and a half max. Nobody’s paid to do it, they don’t have weeks to spend.
But beyond that, peer review in climate science has become clique-ish and inbred, with people giving their friends papers a passing grade without even asking if the data is archived. We should be long beyond that now. But no, the latest Wahl/Amman “independent reconstruction” could set a new record in that regard. There’s a discussion of the paper at ClimateAudit where Ross McKitrick, the other author of the paper with Steve McIntyre that showed the Hockeystick was built on sand, says (my emphasis):
Go and read through the citations I sent. There is an entire discussion in there by Wegman regarding the pathetic state of peer review in climate science, along with a discussion of the lack of independence of the “independent confirmations” of Mann’s Hockeystick. There is a detailed description of the complete lack of anything resembling peer review of the Wahl/Amman paper in the Bishop Hill cite. Did you not read them, or did you read them and pretend they weren’t true, or what? If that’s the level you want to play at, I’m not interested. Take a lesson from jshore. Read the cited works. “Peer review” in climate science is all too often a tragic joke. The Hockeystick was peer reviewed … and the Wahl and Amman paper was shopped to different reviewers until they found somebody to look the other way.
I hate to say it, my friend, but you’ve been duped. Peer review is not a Seal of Good Housekeeping. Heck, it’s not even a seal that they’ve archived the data, and how shoddy is that? Wahl and Amman’s data was cited (against the IPCCs own rules) in the IPCC FAR, but they refused to archive the Supplementary Information until a couple weeks ago … and now that they have finally come clean, we can see why they waited so long. It’s also clear why they had to shop for reviewers. It’s because the SI shows the main claim of the paper is false. It agrees with the analysis of McIntyre and McKitrick, that the Hockeystick claim of significance is false.
Steve McIntyre’s comment on this fiasco was
You may wish to depend on that sort of “peer review”. I do not.
For those wishing a good overview of the early history of the Hockeystick scandal, there was an in-depth article in the Dutch magazine “Natuurwetenschap & Techniek” (Science and Technology) in February of 2005 that is well-researched and highly readable.
Yep. And ignoring what would clearly–to someone in the field–be a significant factor in your study would be an error. Describing your methodology insufficiently to reproduce it would be an error. Etc. I didn’t say anything else, and in fact specifically said that any one study is assumed to be inaccurate.
Your cite for this is one anecdote as told by the petulant whiner himself? :dubious: Find a survey with a significant percentage of a decent sample size of climate scientists saying that they feel they wouldn’t be able to get published in a respectable paper if they had a quality study with good data due to some sort of politics or what-have-you and we’ll talk about it. I mean, at least try to prove your point.
Wouldn’t be able to get published? I have not said a word about being unable to get published, not one, nor did Ross McKitrick in the piece I quoted. Why are you talking about a survey? What do I care? That has nothing to do with the subject.
Ross McKitrick was talking about the Wahl and Amman paper, the Jesus Paper, and why it matters. And it’s not just that one cite. I’ve given you a number of cites on this very paper, and a number more on the general subject. They show a host of problems across the field of climate science. Sounds like you only read one.
Look. The Hockeystick has been discredited in the peer review literature. Last publication was McIntyre’s Reply to Huybers in GRL, which has never been refuted, by Huybers or anyone else, unless you know something I don’t. Hockeystick was bad math. Top US statisticians called it an error. Bad proxies too. No significance.
However, it was also the logo of the IPCC Third Assessment Report. It was the only graphic to make it into the Summary for Policymakers. It was reproduced all over the world. Al Gore foolishly used it in his movie, and didn’t even know it. He called it “Dr. Thompson’s thermometer” and claimed it verified the Hockeystick, when it was the Hockeystick …
Bad math. Bad proxies. Meaningless. And all of the so-called “independent verifications” include the bristlecone pines as a proxy. If you throw bristlecone pines into the proxy mix, you can get a hockeystick by simple linear modeling, no need for eigenvectors like Mann used … so what?
The most disturbing part is this. Mann knew the hockeystick depended almost entirely on the bristlecones (plus a small bit from the Gaspe series). He knew it before he published. He claimed it was “robust”, when he knew it all hung on one proxy, a single site in the southwestern US. He knew it, and he PUBLISHED THAT PACK OF LIES ANYHOW!
… ‘Scuse me for shouting, but as I have said before, I’m a reformed cowboy almost all the time. But if I ran the IPCC, I’d fire that logo-totin’ scientist’s ass right there and then, and institute a wide ranging look at the whole IPCC process to make sure I never got fooled again. I mean, the IPCC, it’s supposed to be the real deal, the hard science, things we’re surest of, our best and firmest knowledge – not some farrago a scientist just cooked up in his computer.
Wahl and Amman’s paper, the Jesus paper, the one that the IPCC flouted their own rules to take aboard for the Fourth Assessment Report (FAR), is the latest incarnation of this nonsense. A year or so ago, W&A claimed they had shown Mann was right. Then W&A didn’t publish their long-promised “Supplementary Information” that showed their method in detail. With the SI not published, nobody could determine if they were correct. Finally, they were finally pressured into publishing. And guess what?
They had lied.
Despite all of their public statements that they had finally salvaged the Hockeystick, despite statements depended on in the FAR, despite all their false claims, their results agreed with Steve McIntyre’s. The Hockeystick is busted, and their results showed it … but unfortunately, the system that spawned it lives on. Why am I telling you this? Read the Bishop Hill piece, he tells it all, and better than I could.
Did the IPCC reform its policies to ensure that junk science didn’t make it into the next report, the FAR that was just published? No way. No mention of the fact that their logo blew up in their face. No mention of the Hockeystick in the entire FAR, actually … yeah, that’s how grownups handle system failure, they whistle, look at the sky, don’t mention the name of the Departed …
The citations I’ve posted involve a wide range of players including the IPCC, Nature and Science magazine’s refusal to enforce their own archiving policies, peer review and the quality thereof, the ungodly disconnect between the climate science community and the statistical community, the NSF not requiring adherence to their own rules. I’ve given you the words of various people who were among the participants. I’ve given you words of authors of the papers under discussion. I’ve given you statisticians. Heck, I’ve even given you Greek hydrologist statisticians, what more could a man want? …
And you call that, all of those citations about the scientific malfeasance occurring in the name of climate science, all those different references explaining various aspects of the problem, you want to reduce all that to one “petulant whiner”?!? And then you think that what Ross McKitrick is talking about is whether it’s hard for skeptics to get published?
I gotta say, his fighting ignorance is harder than I ever imagined …
Well, let me point out some of the statements that the NAS report makes and see if you and McIntyre would agree with them. If you agree with these statements, then I don’t think we have much to argue about:
(bolding added)
The simple fact is that Wegman was picked by the Republican majority of the committee and that he was given a very narrow charge, i.e., to look at the statistical method actually used by Mann et al, which most people now agree is not the recommended method of doing this…and not at the larger question of whether it makes any difference to the final result. (In fact, he only could be given a fairly narrow charge given that he wasn’t really qualified to look at the larger picture being that he has no background that I know of in climate science.)
I didn’t say that I thought his opinions were for sale (and, in fact, as I understand it, he did not get any monetary compensation for his report). What I think is that they picked a person and formulated the charge that they gave him in such a manner that they knew they were likely to get the answer that they wanted to get from him.
Furthermore, despite not having a background in climate science and demonstrating significant ignorance of it at the Congressional hearings, subsequent to issuing his report Wegman nonetheless has seen himself as qualified enough to sign on to this letter basically at odds with all the major conclusions in the field. And, yes, if I saw him, I would ask him to his face why he signed that letter and how he felt qualified to agree with what the letter stated.
Furthermore, his analysis of the paleoclimate field and the connections between researchers was sort of silly…and basically just showed what most people already know, which is that in a small field, a prominent person in that field is likely to have worked with a larger number of the other players in that field.
I am not saying these other people are perfect or that McIntyre and Wegman and you are playing fast and loose with the truth. What I am saying is that I am going to trust sources like the NAS to give an unbiased view rather than sources like you and McIntyre and Wegman.
It might include that on your planet and yet you were unable to appreciate the error in using appropriate procedures that Douglass et al. made even after having it pointed out to you. You were also apparently unable to spot the basic problem with the McKitrick paper on global temperature.
And, on the other side of the fence, you have consistently been unable to demonstrate that you understand what the Santer et al. even says, preferring to present a “strawman” version of what it says.
I agree that a version of Mann et al.'s result (although I don’t think from the paper that you are focussing on, as discussed below) was shown prominently in the report but you will have to clarify what you mean by your two claims here. It does not appear to be the front cover logo of any of the TAR reports and I don’t know what you mean by it being the only graphic in the summary for policymakers.
When Mann et al. wrote another, longer paper the next year (that also extended their results back further than the original paper), they titled it “Northern Hemisphere Temperatures During the Past Millennium: Inferences, Uncertainties, and Limitations” and they stated clearly that in order to get good verification, the proxy data from that site in the southwestern US:
When I have pointed this out before, you have cryptically said that they are talking about a different proxy data set without elaborating or bothering to explain what the differences are and how what they said in that paper relates. (Admittedly, here Mann et al are talking here about the reconstruction going back all the way to 1000 AD rather than just 1400 AD, as in the first paper. However, I believe that this graphic going back to 1000 AD is the one that the IPCC actually used in its report both as a figure and to reach the conclusions that it reached, so it is really the most relevant one to discuss. Confusingly, you seem to mix the two up when it suits your purposes but then claim that they are totally different and thus that what Mann et al say here is somehow irrelevant when it suits your purposes.)
The IPCC reviews the status of the current literature. It is meant to be an assessment of the current science based on that literature. It does not verify that each and every paper is correct. And, it does not claim that everything that they conclude based on the literature is known with certainty. In fact, in the TAR, the conclusions based (in large part) on the Mann et al. work were stated as being “likely” (>66% confidence). And, the IPCC AR4 has re-affirmed the main conclusion that they said was “likely” from the previous report regarding the warmth of the late 20th century.
The “various people who were among the participants” that you have given us the words of are all on one side of this debate. If by the “Greek hydrologist statisticians”, you are referring to the severely-flawed Koutsoyiannis paper, I have no idea why you are bringing this up since it has absolutely nothing to do with the Hockey Stick.
Yeah, I did see that but I thought it was a pretty weak response. For example, he concludes that the answer to the question posed in his paper was not obvious ahead of time by noting that entering his paper into google yields lots of hits…to blogs and so forth. Brilliant logic!
Overall, I am not particularly impressed by that response.