The citations I sent were not about “the idea of sensitive dependence to initial conditions, chaos, turbulence, and such”, so it is not clear why you are discussing that question. Please read the citations.
You are confusing models with reality when you say “the basic response of the climate to a perturbation such an increase in the concentrations of greenhouse gases is quite robust.” What you mean is that the models generally agree with each other … is that supposed to impress me? Since (as I have repeatedly pointed out) the models use very different forcings, they should give very different answers. The fact that they agree means only that they are tuned to reproduce the past climate … like the song says, “that don’t impress me much”.
This is nonsense and you know it. Climate models and the discussion of climate are about anomalies, not absolute temperatures. If you have a model that can tell me if the anomaly for next March will be larger than the anomaly for next September, let me know. Until then, give this argument up, it means nothing.
Contrary to your claim, validation and verification of models and simulations is well known, applied, and studied in the physical sciences. It even has its own name, “M&S V&V”. NASA applies it to, inter alia, such physical models as those dealing with spatial response of temperatures at the edges of surface discontinuities, positional accuracy, thermal radiometry, atmospheric radiometry, full-sky imaging, hyperspectral radiometry, and bidirectional reflectance. Here is a paper on “Implications of M&S Foundations for the V&V of Large Scale Complex Simulation Models” … I could give you many more examples of the use of V&V for models of physical systems, or you could find them yourself.
You keep asking why I should not accept the judgement of the consensus of scientists? Your entirely specious claim, that V&V is not used for models in the physical sciences, is a perfect example of why I don’t. You put it out there as though it were verified scientific fact, based on your experience as a scientist, and yet it is 100% wrong …
Yes … and you have clearly spoken as a scientist about how physical models don’t need or receive V&V …
We’ve been over this ground many times, jshore, and I have laid out for you exactly what I think we should do, starting with performing V&V and SQA on the models and moving on from there. Your claim is a straw man, and your pretense that I have not laid out my thoughts in detail does not become you.
Say what? I claim models are tuned to reproduce the past climate record. Are you seriously claiming that the models are not so tuned? Heck, even Gavin Schmidt admits that they are tuned. Or perhaps you think they are tuned so as not to reproduce the past record?
jshore, a further thought about tuning the climate models to match reality, which you say doesn’t happen. In all of the climate models, both the clouds and the albedo are parameterized. In the GISSE model, the cloud coverage is low by about 11% (58% modeled, 69% measured). Now the albedo is a function of the amount of cloud cover … but the model gets the albedo just right, about 29.5%.
How do they do that?
To quote Gavin Schmidt (ibid),
Perhaps you might want to reconsider your claim that the models are not tuned to match reality …
I did not say that, I said you are not aware of what NASA is doing with V&V, and it is not only for the Space Shuttle, the main point was that the focus of the testing in models is the physics and the chemistry, checking the coding is important, but it is not the only reason why you come to trust a model.
V&V is not used* much in climate modeling, but there are other reasons why it is so, and on the documents from NASA I see that the V&V department sees not much of a reason to check the climate models, but thinks the data going to them is the most important thing.
(*but logic tells me that it is being implemented in more areas as computer power increases)
Now, even though is clear V&V is being used by climate researchers in the collecting of the data, why not do this before with climate models? I do not think one should jump in the rapidly shrinking anti-global warming bandwagon, and assume nefarious ideals from the climate researchers. It seems to me there are more mundane reasons why V&V was not a big deal before with climate researchers.
I agree with jshore:
While you have found a flaw on that **jshore ** said, (that is not accurate to say V&V is not used the physical sciences) It is accurate to say it is not used much in climate modeling, and the more I check, the more I’m convinced the reason is that the testing has been done in other ways and there are more mundane reasons why V&V has not been implemented at large.
What I conclude of all this is that you may have a good point, V&V is important; however, it is clear that it is ignorant to claim no testing has been done, as there are other ways to test. V&V is there to check the coding, not the science involved. It is clear to me cost was one big reason why it was not implemented yet in many climate models; however, it is clear also that organizations like NASA have been involved on it and it is beginning to show in their climate research.
And yet, since V&V deals with the quality of the coding I have to conclude that applying it at large in climate models will change the results very little.
For as it has been pointed before, computer power is increasing for everyone, even skeptics. Seeing that climate models have been here for decades it is important to notice that that critics would already have shown not only the flaws in past models that were declared good by climate scientists and physisists, but also show that the corrected models have definitive contradictory evidence. Instead we recently got errors that, as much as the extreme right media tried to puff up as a way to discredit the research, turned to still confirm the overall findings of today’s climate researchers.
Actually I think it’s the warmers who are trying to change the rules by replacing bedrock concepts like prediction, disclosure, and reproducibility with vaticinium ex eventu, proprietary software, and a “trust us, we’re scientists” attitude.
In fact, I did and the citation you wanted me to read (here) starts out with the quote: “If a time dependent equation has a solution that grows exponentially in time, then that solution is very sensitive to errors in the initial condition, i.e., any error in the initial condition will cause an exponential deviation in time from the true solution.” Perhaps you meant to link a different cite?
Well, then take the difference between the March temperature and the average temperature and call it a seasonal anomaly. My point is simply that predicting weather and climate are two different things and I illustrated this using the example of a climate prediction based on the seasonal cycle.
Basically, what GIGObuster said (and most especially, the quotes from Tim McDermott who frames the issue much better than I have). I will admit that my original statement was too sweeping. And, yes, there do seem to be some examples in the physical sciences where the decision has been made to jump through these hoops. And, perhaps for the particular application, it was the right decision. However, I contend that this sort of formal process is still most useful for certain applications where the scope of the mission that the code is supposed to perform is narrowly defined and quite easily testable. It is not well-suited for science at the forefront of our knowledge where more creative testing and less formalism is required.
As you yourself noted, climate modeling is very complex and noone is contending that climate models are anything but approximations to reality. The best way for scientists to determine the extent to which they think their models are getting certain features right is to leave the testing in their hands rather than put it into the hands of a bureaucracy. And to encourage scientists to do lots of intercomparisons among the models and such. I think it might be useful for funding agencies to consider how best to facilitate scientists testing and intercomparing models. However, I think it is counterproductive for them and others to dictate upon them a formalism and bureaucracy that is ill-suited for the task.
Okay…I think I need to clarify what I (and presumably Gavin) are saying here. I am not claiming that there is no tuning of parameters whatsoever. I have said before that there is. However, I am saying that this tuning is not to reproduce the past global temperature history but rather to get some basic things right. One of those basic things is, of course, having the system be in radiative balance when it reproduces a reasonable climatology and in the absence of variations in external forcings. It is obvious that if you are not getting the right radiative effects down to very high precision (as you have correctly noted they are not) then you will get drift in the climate because of this lack of radiative balance. So, yes, it is not surprising that they have to tune some parameter to get that.
However, my contention (well, not just mine, but the climate science community who has studied this) is that there is no alternate way they can find to do this tuning that would reproduce the past record in the absence of including anthropogenic forcings. In particular, if you just use the known natural forcings, you are not going to be able to see the sort of temperature rise that we have seen over the past ~35 years because the natural forcings simply do not have the right time dependence to produce this. It is only by including anthropogenic forcings that you are able to produce this. So, if one believes it is not due to anthropogenic effects, one has to come up with another natural forcing mechanism that has been missed by the scientists and has the right time dependence…and one also has to find a way that the climate model has a significantly lower sensitivity to the known forcing due to greenhouse gases.
Admittedly, in an ideal world, one could calculate everything from first principles and could do so with such precision that all the absolute errors were well less than the forcings of interest. However, science at the forefront of our knowledge seldom exists in such an ideal world…It exists in the real world where nature challenges us to work harder and we have to assemble a large array of evidence because no one test or model is ever alone going to be definitive. Welcome to the real world of science.
Well, I would say that the National Academy of Sciences (or the Royal Society or …) might be in a better position to judge such things than, say, some non-scientist or amateur scientist on a messageboard. Why do you think that they don’t seem to agree with you?
GIGObuster, thanks for your post. You seem to be saying that V&V costs to much to do. I wasn’t very reassured by your quote from Tim McDermott, who said:
One would imagine that if the NAS agreed with you, they would be decrying such a horrible distortion of the scientific process rather than speaking out strongly, in concert with their counterparts in other countries that the science on AGW is certain enough that it is time to take action.
My apologies for my lack of clarity. We were discussing the question of whether increasing the resolution would improve the forecasts.
The citations discuss the difficulty, perhaps insoluble, of the way that the computer models address the problem of “the idea of sensitive dependence to initial conditions, chaos, turbulence, and such” that you mention. They were not about the problem, which is what you had were referring to. They were about how the solution used by the modelers (an artificially increased turbulence) does not work, as evidenced by the fact that when you increase the resolution, the “solution” no longer works.
Let me see if I can make it clear why your argument is meaningless.
Your argument is
Temperature is cyclical on two scales, daily and annual. Replacing your annual cycle with the daily cycle, we can say:
“Intuitively, you know that while you would never trust me to predict what the weather will be 5 months from now here in Rochester, you would trust the prediction that 2AM have a considerably lower average temperature than 3PM.”
Yes, but … so what? We’re not trying to predict whether afternoon is warmer than night time, or whether September is warmer than February. The fact that we can say that in seven years nighttime will still be colder than the afternoon, and September will still be warmer than February, proves nothing about predicting either the climate or the weather.
jshore, V&V is designed to make sure the code is correct, the input data is correct, and the various algorithms actually work. This is not an undue, onerous task, it’s not rocket science, it’s a necessity. Having correct code and data is not an option, it is a requirement, even in the simplest applications, as James Hansen recently found out to his cost.
You seem to think that the best person to put in charge of the chicken house is the fox … in other words, you think the best people to check and verify the calculations are the scientists that did the calculations.
But why should the person who made the error be the best person to find the error, when they didn’t notice the error when they made it? In fact, the person who originally did the work is the worst person to check the work. That’s why newspapers have proof readers, because it’s too easy for the original author to overlook the same error twice. And as the recent experience with Hansen showed, your recommended process (let the scientists do it) has proven inadequate to find even the simplest errors.
In addition, Hansen has repeatedly refused to reveal the code that he uses for the simplest of tasks, that of creating the global temperature average. Why not reveal it? While you might trust a very opinionated man who won’t show his code to verify that same code … me, I don’t. Nor is he unique. I’ve been unable to get Phil Jones’ HadCRUT data, even with a Freedom of Information request. Mann, and Osborne, and Briffa, have all refused to show their data. Thompson won’t reveal his glacier core numbers, the list goes on and on. A six year old child might trust those folks to make sure their own code is correct, but here in the real world of science, reasonable people expect more than that.
In addition, there is an open question as to whether the various approximations used in the models actually converge to the right answer. This question is discussed in the citations I gave above. Perhaps you trust that the scientists running the models have verified that their code converges. I don’t. In the main, they don’t even seem to have acknowledged the problem.
Finally, these models are huge, and each one has been worked on by a large number of programmers over a number of years. As a programmer yourself, I’m sure you know that the odds that there are no bugs in the program is vanishingly small.
jshore, if this was just some random model that didn’t have much riding on it, I’d be the last man to ask for V&V and SQA … but it’s not. You are asking us to bet billions of dollars on the model results, but you are extremely unwilling to even consider subjecting the models to the normal, usual testing that mission critical software receives. Instead, you ask us to take it on trust …
Why your reticence? Why not test the models, just as we routinely test other models of equal importance? As far as I’m concerned, with billions of dollars on the line, the need to test the models is a given. If you think they should not be tested, you need to justify that in the strongest possible terms. Saying ‘It’s not usually done’ doesn’t cut it when gigabucks are at stake.
Welcome to the real world of science? Perhaps you could give us one other example of advocates asking us to spend multi-billions of dollars based on the results of untested, disputed, parameterized, poorly understood models of an immensely complex system … I can’t think of one other example of that. That’s not the real world of science, that’s the fantasy world of modelers.
I pointed out elsewhere on SDMB that the GISSE model parameterizes ice-melt pools (liquid water on the surface of the ice) by restricting them to an exact six-month period. This, of course, is not how the world works – in fact, ice-melt pools are not constrained by the calendar.
Now, why did they put in that parameterization? To “get some basic things right”? No, because ice-melt pools by the calendar is not right. They put it in to make the model more accurately reflect reality, as exemplified by the historical record.
You and Gavin say that there is no way to get the models to fit the historical record because they are too complex. But the same process used with the ice-melt pools has been followed from the beginning, starting with the simplest of models. A new forcing is added, and the parameters are adjusted to make the new output fit the record. Then another forcing is added, and the parameters are readjusted.
Another example. The dip in temperatures from ~ 1945-1957 is explained in the models as being from low-level, tropospheric aerosols. We don’t know, however, what the combination of the direct and indirect effect of aerosols is, the level of understanding of this is very low.
So the aerosols are added to the model, and a value is chosen for the aerosol effect. What value is chosen? Yes, you guessed it, the value that makes the aerosols balance out the CO2 and reproduce the historical record.
All this is done despite the facts that:
We don’t know the size of the aerosol effects from actual observations – the estimates of the size of the effects comes from the very climate models in question.
The majority of the aerosols are in the Northern Hemisphere, but the majority of the 1945-1975 cooling did not occur there.
The model estimates for the regions where the aerosols have had the major effects shows no relationship to the actual temperature changes in those regions.
A recent study shows that low level aerosols in Asia result in warming, not cooling.
Given that the majority of the aerosols are in the Northern Hemisphere, we’d expect the Northern Hemisphere land to warm less than the Southern Hemisphere land. This has not happened.
We have very poor data for the historical levels of aerosols, and there is evidence that the particular aerosol levels chosen by the modelers may not represent the true changes in aerosols.
So the aerosol explanation simply doesn’t hold water. It assumes facts not in evidence, contradicts facts in evidence, and requires a careful selection of parameters to make it work. Yes, the models match the past … but whether you like it or not, it is because they are tuned to do so.
A more likely possibility is that the NAS would take the position that the deviations from the best practices are justified or excusable.
But that’s not what you seem to be claiming. You seem to be claiming that according to the NAS, the science supporting AGW does not deviate from the normal best practices of science in general.
Please quote the NAS to back up your claim.
Also, why are you ignoring my earlier question?:
Why do you think that a climate modeler should disclose the results of all tests, whether good or bad? What’s wrong with just disclosing good results and throwing the bad results down the memory hole?
Care to retract this generalization? I can call it when **jshore ** does it, and here you really lose any respect anyone could have for your position. Since my searches also have found evidence that climate models have been tested (and not just with historical data) I have to conclude once again that you are discussing in bad faith.
Well, I suppose that your explanation is possible…but then why would the NAS still feel justified in saying what they said? At the very least, they must think these deviations are not very relevant. However, there is no evidence that such deviations have occurred except for your claims.
At any rate, if you are trying to argue that the NAS might think that your opinion is correct but completely and utterly irrelevant then I guess I would not dispute that as a conceivable interpretation.
I don’t trust James Hansen’s figures, for the obvious reasons. Here is the HadCRUT data. It shows the Southern Hemisphere cooling more than the Northern.
The myriad differences between HadCRUT and GISS, which are the premium global temperature datasets, and which are both based on the same raw data, should give you a clue about the relative maturity of climate science. We don’t even have an established baseline to work from. When you claim that the models reproduce the historical record … exactly which record are you talking about?
Consensus? How can you say we have a consensus? We don’t even have scientific consensus about the past temperature, to say nothing about the future temperature.
It’s not for me to speculate. I pointed out that it was actually “your side” that is trying to change the normal rules of science. You claim the NAS disagrees with me? Fine, quote some actual language that contradicts my point. I’m not interested in your guesswork.
There is plenty of evidence in this very thread. Evidence that climate models are tested on historical (as opposed to future) data and evidence that the testers in many cases refuse to release their source code and/or data which would allow people to check, verify, and reproduce their results.
i.e. trust us, we’re scientists.
By the way, why do you continue to ignore my earlier question?
The evidence so far is that the historical tests are only a part of the testing. And the best you could say about testers not releasing source code is one example were already the data was released. And the checking the critics made of that data showed to me that indeed, the critics looked better when they could made the point the tester was not releasing the data. As it is now, I’m still waiting for them to credibly point to what was the big flaw in the original research. (And with computer power getting accessible as it is nowadays, there is really no excuse for this weak counter argument IMO)
What this last post of your showed is that you are very selective on the evidence presented.
Please describe how AGW has been tested, keeping in mind the gold standard of testing a scientific hypothesis, which is to make interesting predictions about future events.
How many examples of failure to publicize source code and/or data would it take to convince you?
