Thanks, jshore. Actually, the situation is far more complicated than you indicate. See this thread for more details. Among other things, there are University policies as well as senior US government policies on the matter. From that thread, here’s one of them (emphasis mine):
This speaks clearly to the larger issue here than the legalities. Science depends on transparency and replicability. If a piece of research is not replicable without access to the code, then it should not be considered scientifically established. Unfortunately, as with Michael Mann’s “Hockeystick”, certain irreproducible results have gained wide currency as though they were both replicable and replicated, when neither one is true.
So while you and Mann can certainly stand on the strict legal technicalities of the inadequate NSF guidelines, you are turning your faces against science when you do so. I thought you said you were a scientist, but you continue to give me reasons to doubt it.
Turning to some good news, Hansen today bowed to increasing pressure from the scientific community and released the code used to calculate the averages for the global temperature. I took a very quick look at it, and was amazed to find it is not a single program, but a pastiche of no less than 53 separate Python, C, Unix, and Fortran programs! It will take a long while to hack through all of the possible errors in that mess. I’m finally starting to see one reason why they don’t want the code subjected to V&V … it’s downright embarrassing.
That would make sense, if the Met office in England had not checked on its own to agree with the overall picture. The overall trend and evidence were replicated by not using the same code, and that to me is important. It shows that once again it is physics and chemistry what is the basis for the models.
And as some missed, I agreed already that the release of the code was important, lawyers and NASA be dammed. As I see it, when other groups already agree on the results, I have very little worry that the skeptics here will find that the conclusions of the research are wrong.
The coding on the other hand… I know the feeling. Going through other’s code. Not a pretty sight. The annoying thing was to find out in the end that the damn programs still did the work, as my computer teacher said to many students: “you lose points in the coding, not the execution”
Well, it does sound like it gets complicated as to who owns what. But, still I don’t see anything in what you quoted that talks about having to make source code available.
Well, there are a few issues here. First, you and the whole ClimateAudit gang seem to have come up with a novel new standard for replication in the physical sciences whereby you are satisfied with nothing less than being able to reproduce the result down to the last jiggle. As I have told you many times, this is not the standard that is generally applied in the physical sciences where generally the idea is that your follow the basic procedure outlined in the paper and see if you get a results that shares the same important features of the original. In fact, I have challenged you to go through a physics journal and find any paper that has some modeling in it and see if they have given you enough information to do the sort of replication that you seem to want to do. I think you would find such papers to be very few and far between.
Hell, when I referee papers, I find it challenging enough to get the authors to freakin’ explain even the basics of their model. I am continually amazed with what passes for explanation in some people’s mind. So, as you can see, I hardly think I am on the extreme in this regard…In fact, I think that most people who have had me as a referee (or have had me as a co-author) would say I am closer to the extreme of being a stickler.
In general, in my own personal experience as a scientist, I would err on the side of openness. But, to be honest, I have never had any scientist as far as I can remember ask me for my computer code. Of course, this may be that I published on less contentious and ground-breaking issues…but I can cite a couple of cases where people wrote papers who had as pretty much their main thesis the claim that we were wrong (admittedly, in one case, about something that we clearly stated was conjecture), and still these people did not demand that they see my computer code.
Well, yeah, I’ll admit that in giving out code, that is one of the things one has to deal with is the embarrassment! I recently gave someone some MATLAB code within my company and I told him straight-out that I was kind of embarrassed that he gets to see my cheap substitute for an “case…switch” (or is it “switch…case”?) statement which I never bothered to learn how to use so that I instead tend to have lots of
that I use to quickly go in and set the case to “1 == 1” that I actually want to use.
Given an infinite amount of time, I am sure we would all produce wonderously elegant code…but in the real world, we end up cobbling things together. Rather than going through someone’s code of this sort and making lots of mountains out of molehills about each bug that you find, it is often more productive to just do your own analysis independently and see how it compares. This is a much better check anyway.
That’s why I think it is good that NASA GISS and HadCRUT have their independent analyses. (And, no, I guess I don’t exactly find it shocking that one gets a trend of 0.060 C per decade and the other 0.071 C per decade.) And, I am quite sure neither code to produce the temperature averages is absolutely bug-free…but the question is more whether or not the bugs amount to more than a hill of beans.
At present, for those under NSF grants, you are right. As source I cited said:
So at the moment, there is nothing but ethics and scientific integrity that requires making the source code available. Unfortunately, some climate scientists seem to have neither.
You don’t seem to understand the difference between the physical sciences and mathematics. Taking a dataset and subjecting it to a series of mathematical transforms is not physical science. It is math, no matter whether the dataset is random numbers or tree ring widths.
Unlike the physical sciences, unless you think that 2 + 2 = 4.01, close is not good enough in mathematics. This is not something invented by Climate Audit or by me, it’s the nature of math. I challenge you to find any mathematics paper that thinks that 2 + 2 = 4.01 is close enough.
All that Mann did, every bit of it, was math. There were no experiments to be repeated. There was no time in the lab. There were no visits to the field to record data. He simply took a dataset and transformed it. In that case, close enough is not good enough. If we do the same transforms, we should get the identical result. If we don’t, something’s wrong. If you think this is a new idea, something I just made up, you need a quick course in the history of mathematics.
jshore, I’m not questioning that you would err on the side of openness. But in common with most scientists, you are not asking us to bet billions and billions of dollars on your code. If you were … we’d be absolute, inutterable fools not to only ask for the code, but to subject it to V&V and SQA.
So. Two transformed datasets, purporting to calculate the exact same thing, are produced from one source dataset. The results are statistically significantly different, not just in trend, but in individual (monthly) values. In addition, the residuals (A - B) are wildly non-normal. This means that the differences are not just random error.
You don’t find that shocking? You sure that you’re a scientist? I hate to keep asking, but you keep making unscientific statements.
Because any scientist I know would be very curious how we can end up with such widely different answers starting from the same dataset.
You greatly overestimate the differences between the models. From Nature magazine (emphasis mine):
Even the IPCC thinks that the models are too similar to use as independent evidence. That’s why model intercomparisons, such as those in the CMIP, cannot tell us anything about their accuracy or or the spread of their errors.
Well, it is not as clear to me as it is to you that either ethics or scientific integrity requires them to do so.
Look, my whole freakin’ dissertation matches what you describe above. It was simply mathematics…but it was not in the field of mathematics. It was in the field of physics. And, all I can tell you is that in the field of physics, even if what you are doing involves only mathematics, people do not generally go around trying to replicate other people’s work down to the mathematical precision on the computer. It is simply not the way it is generally done.
I think one of the things that probably really pisses climate scientists off about you ClimateAudit folks is when you come into the field and pretend like you know how science is done in other closely allied fields (like physics) when you haven’t worked in those fields and you don’t in fact know. [I know there are a people at ClimateAudit, like David Douglass, who are actually trained and working in the physical sciences but I get the impression that he is the exception rather than the rule.]
You are perfectly entitled to think that it is the way it ought to be done but don’t go around trying to claim that it is in the physical sciences, because I can tell you that at least in physics it is not.
Well, first of all, I am not convinced they use exactly the same station data. Sure, they get their data from the same sources but they presumably cull and select in in different ways. Second, they have to do a lot of different things to the data…They have to try to weed out bad data, they have to try to interpolate the anomalies in order to get an average over the sphere that is the earth, and so on and so forth. Of course there are going to be differences…and very likely some of these differences will be systematic. The question becomes whether they are important enough that we really need to know the global temperature changes to a higher degree of accuracy than they are giving it to us. Of course, it would be ideal to know everything to infinite accuracy…but is it necessary?
It never ceases to amaze me what an idealistic fantasy view of science some people seem to have. Real science is a lot grungier and grimier than it is presented in the textbooks…especially the ones that someone reads at the undergraduate or lower level. The number of things that can be solved exactly and with full rigor is vanishingly small…and such systems are usually quite idealistic models that bear only a very rough relation to any real world systems.
Along the lines of my last point: Actually, the recent paper by McKitrick about the global temperature not being able to be rigorously thermodynamically defined was a perfect example of this kind of thinking. I mean, I think all but the most theoretical physicist or mathematician would be like “Well sure, but who cares?” The practical question is whether there is a way to define it that makes reasonable sense…Or whether any practically reasonable definition will really get you a very different result for the temperature trend. Sure, if you do some totally f-upped average where you take the temperatures to the 200th power before you sum them and take the result to the 1/200th power then you can get some very different trend result…but since you are just selecting out the highest temperature in the data set in that case, what sort of reasonable person would choose such a dumb averaging method?!?
Since the UK Met Office model (HadCM3) was used in the IPCC report, and since the Nature quote (that checking one model with another was akin to buying a second copy of a newspaper to check the facts in the first copy) referred to all of the IPCC models, I fear the shoe is on the other foot - you need to show that the British used a model that was not used by the IPCC.
GIGObuster, I begin to despair. The Indian Monsoon is not an “extreme weather phenomenon” as you claim, it is one of the largest, most regular regional-scale features of the climate. Gavin Schmidt claims that these large features are captured in the GCMs:
The Monsoon has a huge spatial scale, covering most of both India and the Indian Ocean. It is not on a “very local scale”. It is also quite regular, in contrast to the ENSO and the NAO. So the GCMs should have a “reasonable representation” of the Monsoon.
But despite the millions of dollars and thousands of man-hours that have been put into the climate models, the 1932 Monsoon predictions are better than any predictions of the GCMs.
We were discussing testing the GCMs … there’s one test. They failed. I could provide you a host of other scientific papers detailing the shortcomings of the models … but since you think that the Indian Monsoon is an “extreme weather phenomenon”, I’m not sure it would do much good. I doubt that I could shake your faith with mere facts.
That is not what the modelers claim to do accurately (yet), and I have seen that in several reports, even from the Met office (what is clear now is that you did not bother to check), this is just once again an example of your bad faith discussion tactics.
I didn’t “lump the british model” in with anything. I cited a Nature article that referred to all the models used by the IPCC, which includes both British and US models.
I quoted Gavin Schmidt, a noted modeler who is one of the main people behind the GISSE models, as saying “Most GCMs are able to provide a reasonable representation of regional climatic features …” If you have a problem with that, take it up with Gavin, not me. He’s the one making the claim of reasonable representation of regional climate features.
A reasonable representation should give reasonable results for a regional climatic feature like the Indian monsoon. I showed that the models in fact give a very poor representation of the monsoon, so poor that better predictions were available in 1932.
Nor is Gavin the only one to claim that the GCMs can forecast the monsoons. Here is a paper (on the East Asian monsoon) that says “The strength of the monsoonal circulation increases in 2×CO2 conditions, leading to a general increase in precipitation over all regions by 10–30%.”
And here is an article that says "Scientists at the University of Liverpool are investigating the anticipated effects of climate change on India’s monsoon season and the impact that alterations in India’s water cycle will have on the country’s people, agriculture and wildlife. Changes to India’s annual monsoon are expected to result in severe droughts and intense flooding in parts of India. Scientists predict that by the end of the century the country will experience a 3 to 5ÚC temperature increase and a 20% rise in all summer monsoon rainfall.
Here is a study of the use of a GCM to forecast the Indian monsoon that says:
“The ability of the current generation of climate models, in their long-term simulations, to replicate the observed atmospheric behaviour on a wide range of spatial and time scales provides support in applying these models to the greenhouse gas-induced climate change projections on regional scales.”
So yes, in fact, a wide variety of people are claiming that the GCMs can be used to forecast the Indian Monsoon. A google search on the subject yields over 15,000 hits … which, to use your words, you “did not bother to check”.
So what is it? Are they a failure or not? Or are you now conceding that they are reliable? :dubious:
I do take into consideration what the Met office said, the resolution is still not here yet to predict local wheater reliably and that is confirmed by the failure of many models to deal with hurricanes for example, when other researchers claim they are able to use models to predict monsoons and **if they can do so repeatedly ** (oh look! a test!) it actually helps my position more than yours, because it means they are getting better at the local resolutions.
And yes, the original point is confirmed, they do test those things. The failure of several models is part of the progress.
GIGObuster, I’ll repeat this first part just once more. I have said repeatedly that the models have not been tested using V&V and SQA. I have not said that they have not been tested. In fact, I have pointed out a test, prediction of the monsoon precipitation. They failed, they did worse than models from 1932. Why is it so hard for you to understand that?
Next, the Nature study showed that the models were not independent, so they can’t be used to check each other as you seem to think. You believe that since the people who produce the models say that they are all just peachy keen, the question is settled … right.
Finally, you keep bringing up proofs that the world is warming … why? Everyone agrees that the world is warming, and has been for the last three centuries or so. That’s not the question, so stop with the proofs that it’s warming, already. We know that.
The questions are:
how much of the warming is due to humans, and
what is the mechanism for whatever percentage of the warming is caused by humans? Land use changes? Black carbon on the snow? Brown haze over Asia? CO2? Methane from ruminants? Change from forest to agriculture? Changes in cosmic rays? Irrigation increasing the humidity? Increased dust from construction and agriculture? Some combination of the above?, and
if we are causing some amount of warming, is this a danger? (Warming since the Little Ice Age has been a net benefit to humans, too cold kills more people than too warm) and
if we find we are causing a significant amount of the warming, and if we can figure out exactly what mechanism is causing the majority of the anthropogenic portion of the warming, and if we decide that (unlike warming in the past) the costs will outweigh the benefits, is there a cost-effective way to reduce the effect we are having?
Those are the questions, and to date, we don’t know the answers. The difficulty is, we don’t know how the climate works. We have no general theory of climate, like we have a theory of gravity or fundamental particles. We regularly discover new, unknown forcings. Plankton, for example, cause clouds … who knew? Given the size of the ocean, is this a significant forcing? We don’t know. Heck for some forcings, we don’t even know the sign of the effect, much less the size.
Take aerosols. What are we talking about? Well, sea spray, and sulfates, and black carbon, and organic carbon, and nitrates, and biogenic aerosols, and mineral dust … the effect of all of these on clouds is very poorly understood, we lack both observational data and a theoretical framework. The models do a very poor job of modeling these effects:
So, given all of those problems with simulating the aerosol/cloud interactions, how do we estimate the effect of aerosols on radiative forcing? We adjust the postulated forcing to best agree with the historical temperature record, and then take the modeled forcing as representing reality … I’m sure you can see the various problems with this approach. At least, I hope you can.
The ugly reality is that we lack both a theoretical framework and observational data for many, perhaps most, aspects of the climate. This is not surprising, since the climate system is unimaginably complex, and goes on at all scales from the molecular level to planetwide. That’s why the models perform so poorly on anything but replicating the historical temperature trends that they are tuned for – because, as the IPCC FAR notes, modeling from first principles is difficult, so we have to guess at the values, and adjust the parameters, until the models are somewhat like the reality.
Which is fine, and there are things to learn from that. But the climate is an interconnect system, so if there is an error in any one of the myriad processes going on in the climate, the other guesses and parameters that we have made will be wrong. We tweak the threshold relative humidity parameter to get the modeled albedo right, and as a result, the cloud cover is wrong. Which means that our parameters for the cloud/albedo interations will be off. Which mean that … and so on.
Consider the “missing carbon sink” problem. When we try to model the carbon cycle, we find that there is a whole lot of carbon that is leaving the atmosphere but we don’t know where it is going. There have been lots of guesses where it ends up, but we don’t know. Now, when we model the climate, we have to assume that the carbon goes somewhere … so we figure it goes into the soil, maybe, or into the plants, maybe, or into the ocean, maybe … but any one of these assumptions creates changes in all of the other parts of the model. So we adjust various parameters until the model works … but does that mean we’ve made the right assumption? Does that indicate which one of these assumptions is right?
We don’t know. You should practice saying that. It’s one of the most important statements that a scientist can make, and is applicable to far more of the climate system than we would like. For example, why does the RSS data currently show that the tropospheric temperature has been falling for the last five years, while the surface temperature seems to be still rising?
We don’t know. And since the models are not based on physical principles, but on parameterized estimations, they can’t answer the question either – none of them predict that the troposphere would cool as the surface warms. Why do you think that is happening, one cooling while the other warms?
Actually, you are misrepresenting what that paper showed. What it looked at was not the representation of the monsoon but the ability to predict the particular monsoon season several months in advance. This is a somewhat different issue…that lies somewhere in between weather prediction (which we know can’t be done months in advance) and predicting general features of climate and how they react to changes in forcings. It is more akin to predicting in the spring, say, whether we are going to have a hot summer or a cool summer in some particular region. In particular, like a weather prediction, it is an initial condition problem…and so does have the potential to be effective by the sensitivity to initial conditions.
Maybe it does, and maybe it doesn’t . . but your apparent viewpoint – that a scientist who builds on the work of another is a “parasite,” is depressing (but not surprising.)
Having other people cite your work is also very positive for one’s career. And there’s nothing stopping you from innovating further.
I think that you’re getting closer to the real explanation here. If you release your source code, people might ask you time-consuming questions like “How come you put this variable in degrees when the function requires input in radians?”
Fine. Since you have been evading the question for pages and pages now, I will answer it for you.
If a climate modeler reports successes but not failures in comparing models to historical data, then in effect, he or she is tuning the model to the data he or she is supposed to be checking it against. Perhaps it’s not a very fine tune, but it’s a tune nonetheless.
Do you agree? Or do you think there is some other reason why these sorts of failures should be reported? If so, what is it?