How does Prof Jones revelation affect AGW in particular and the peer review process generally?

[Quartz]

@Hero_From_Sector_7G

Yeah, time to rethink that empirical approach nonsense.

Umm… no. This discussion is about AGW and the peer review process, not any individual paper. The Scientific Method still rules.

[GIGObuster]

@Quartz

Sorry, but it’s entirely true. He sent such emails. (See his admission about sending awful emails in that article).

Scientists did not make those repeated demands, most if not all where from the same people that do not bother to publish in reputed scientific journals.

[Colibri]

@jshore

I have refereed over 80 papers for the Physical Review journals (Physical Review Letters, Physical Review B, and Physical Review E). I have also published about 30 papers in these and other physics journals.

In no case have I ever asked or been asked for raw data or computer codes. (As a referee, I have sometimes asked that authors explain their methodology a little better in the manuscript…or have asked them for some clarifications about aspects of the methodology that are unclear to me. And, I am quite sure that Prof. Jones has probably been requested to do that too by referees.)

I also regularly review scientific papers for journal publication, and have published such papers. I also have neither asked for, or been asked for, raw data. It is just not typically done. In some cases I have been asked to provide more detail to back up statements, but that’s about it.

The OP is based on a fundamental misunderstanding of the peer-review process.

[Euphonious_Polemic]

@Magiver

Is the modeling and data available for review?

Oh, I see. If you don’t send out your raw data to hostile members of the general public who have an agenda to crap on your research and don’t themselves publish in the field, then you’re automatically guilty of fraudulent behaviour.

Gotchya.

[GIGObuster]

@Magiver

Is the modeling and data available for review?

As mentioned before in another thread, Jones deserves egg in his face for not being good with saving data, however the science is not affected because his research was later confirmed independently.

Hacked climate emails: Phil Jones admits loss of weather data was 'not...

“the science still holds up”. A subsequent study confirmed the original conclusion - that the global warming trend was not significantly affected by urbanisation - and showed that the precise location of the weather stations was not important.

RealClimate: On replication

Of course, this does fit with a point that has been many many times before by the climate researchers at RalClimate: Scientific papers are there so others can (and should) confirm the research, the data that a researcher uses is not really very useful when it can be obtained elsewhere.

Bottom line

Despite minor errors in the printed description of what was done and no online code or data, my replication of the dLM07 analysis and it’s application to new situations was more thorough than I was able to do with MM07 despite their more complete online materials. Precisely because I recreated the essential tools myself, I was able to explore the sensitivity of the dLM07 results to all of the factors I thought important. While I did replicate the MM07 analysis, the fact that I was dependent on their initial economic data collation means that some potentially important sensitivities did not get explored. In neither case was replication trivial, though neither was it particularly arduous. In both cases there was enough information to scientifically replicate the results despite very different approaches to archiving. I consider that both sets of authors clearly met their responsibilities to the scientific community to have their work be reproducible.

However, the bigger point is that reproducibility of an analysis does not imply correctness of the conclusions. This is something that many scientists clearly appreciate, and probably lies at the bottom of the community’s slow uptake of online archiving standards since they mostly aren’t necessary for demonstrating scientific robustness (as in these cases for instance). In some sense, it is a good solution to a unimportant problem. For non-scientists, this point of view is not necessarily shared, and there is often an explicit link made between any flaw in a code or description however minor and the dismissal of a result. However, it is not until the “does it matter?” question has been fully answered that any conclusion is warranted. The unsatisfying part of many online replication attempts is that this question is rarely explored.

To conclude? Ease of replicability does not correlate to the quality of the scientific result.

[Quartz]

@Colibri

The OP is based on a fundamental misunderstanding of the peer-review process.

Agreed, but whose fault is that? We’re constantly being told that the science has been peer-reviewed, as if it has been checked and that is an end to the matter.

Further, from my OP:

@Quartz

After all, peer review is supposed to mean that the science has passed muster with other scientists, and is held as being a higher standard. Yet how can we, the public, respect that when we’re told that it wasn’t really checked at all?

[GIGObuster]

@Quartz

Umm… no. This discussion is about AGW and the peer review process, not any individual paper. The Scientific Method still rules.

Uh, I think he was saying that you are ignoring that science has a long history of living with that “revelation”. Sure, modern technology is now making not sharing the data a moot point, but the fact remains that even when a scientist like Mann does the “correct” thing and releases all his data and codes, he is still tarred and feathered by the denier media.

It is no wonder that (and this was already reported in a previous thread) even the office in charge of the freedom of information requests agreed more often than not with the scientists in declaring that most of the requests were frivolous or baseless.

So far, Phil Jones could be bitten for being so obtuse regarding some of the requests, but I still think that he will survive, the science? Yep, it still rules even if Dr Jones bites the dust.

[Quartz]

@Euphonious_Polemic

Oh, I see. If you don’t send out your raw data to hostile members of the general public who have an agenda to crap on your research and don’t themselves publish in the field, then you’re automatically guilty of fraudulent behaviour.

If your science can’t withstand hostile scrutiny, is it good science?

[Enginerd]

In this case, the “hostile scrutiny” doesn’t play by the same rules that science does. The science is indeed still science; the attacks on it are not.

[GIGObuster]

@Quartz

Agreed, but whose fault is that?

Oh I don’t know…

I guess all that misinformation out there should not be blamed huh?

@Quartz

We’re constantly being told that the science has been peer-reviewed, as if it has been checked and that is an end to the matter.

Further, from my OP:

As it should had been clear from previous discussions:

Yes, they peer review him.

Yes, he was checked by further research.

But, of course it is not the end of the matter, more research is always needed to get better predictions for the future.

[GIGObuster]

For the confirmation and an already posted reply from a **previous **bad article from the Guardian too:

http://www.uea.ac.uk/mac/comm/media/press/CRUstatements/guardianstatement

1. The FOI request was responded to in full

The FOI request from Douglas Keenan was responded to by the university in full in 2007. The data used in the 1990 paper were indeed sent to Mr Keenan, including both the locations of the stations and the station temperature data for China, Australia and western parts of the former Soviet Union. For China, the data covered the period 1954 to 1983. The data were also uploaded onto the Climatic Research Unit (CRU) website.

2. The accuracy of the data and results was confirmed in a later paper

Prof Jones embarked on a study in 2007 which was published in the Journal of Geophysical Research in 2008. In this later study, CRU researchers worked with a Chinese colleague (Dr Q. Li) from the China Meteorological Administration (CMA) in Beijing. Dr Li had been assessing the consistency of 728 Chinese temperature series and his work was published in China in 2007. This improved CMA data was adjusted to account for changes in location of stations.

CRU requested this improved CMA data for the stations that had been used in the 1990 study, and they were incorporated into the 2008 paper.

Figure 6 from this study (see below) shows the comparisons (as anomalies from the 1954-1983 period) between the averages of the 42 rural and 42 urban sites used in 1990 compared with averages from the same stations from the CMA network. The dashed lines are the averages for the rural and urban sites in eastern China from the 1990 paper. The solid lines are the averages from the same stations from the CMA network. It is clear from the graph that the trends of the CMA data for both the rural and urban networks agree almost exactly with the results from the 1990 paper.

I’m beginning to think that Guardian reporter does not know how to use Google.

[Euphonious_Polemic]

@Quartz

If your science can’t withstand hostile scrutiny, is it good science?

It should not have to face hostile scrutiny by people who know little about science, often with an agenda, and funded by an industry that has a vested interest in discrediting the science.

[Hentor_the_Barbarian]

Count me as another example: I have both reviewed others’ manuscripts for publication and published my own papers through the peer review process. I have never asked for anyone else’s raw data nor have I been asked for mine.

At some level, I suppose that abject ignorance of the typical practice of science is our own fault. On the other hand, what the hell am I going to do, spend my time re-running everyone else’s analyses?

[Simplicio]

@Quartz

Agreed, but whose fault is that? We’re constantly being told that the science has been peer-reviewed, as if it has been checked and that is an end to the matter.

I’ve never been told that. Usually when I see “peer-reviewed” come up in internet debates and the like its pointing out that some piece of research hasn’t even been peer-reviewed. Since peer-review is a pretty low bar to get over, that something hasn’t even been published in a journal is a pretty good indication that its some random crackpot rather then a piece of serious research.

Indeed, one can pretty easily come up with a list of peer-reviewed articles whose conclusion later turned out to be wrong. The actual test of a theory is other scientists also publishing their results in their own research that either support or not the theory in question. In many cases this can go back and forth for years or decades, one scientist supports a theory, another publishes evidence its false, back and forth. As long as the papers are interesting and not obviously flawed, they’re all peer-reviewed.

Hell, even the language of scientific articles are keyed to stress that things are tentitive. Results are never said to “prove” anything, they always “support” it. Experiments never “show”, they “suggest”

In other words, I think you’re presenting a fairly flimsy strawman. Ultimately the test of a paper isn’t the Peer Review process, but nature.

[jshore]

@Quartz

Agreed, but whose fault is that? We’re constantly being told that the science has been peer-reviewed, as if it has been checked and that is an end to the matter.

Further, from my OP:

What Simplicio said. You’ve never been told any such thing. No one paper is ever definitive. It is the totality of the peer-reviewed literature that is the best representation of the current science. And, even that is not guaranteed to be correct, but noone has come up with a better alternative than going on what the best current science is telling us.

How can a referee determine if a paper is of sufficient merit without fact-checking the paper?

There is plenty that one can check to determine if it has enough merit to warrant being published so that it can be considered by other scientists: Is the paper clear about what was done, how this goes beyond past work, what conclusions were reached and how these relate to past work? Do the authors seem to have considered major factors that might affect their results? Do the conclusions reached actually follow from the data and analysis presented? And, so on and so forth.

I’m not so sure. Does it not throw out the defence of something having been peer reviewed? Surely it makes the the fact that a paper has been peer-reviewed meaningless? I could write the most utter rubbish and as long as some of my peers say, “That looks interesting”, it will be published.

Indeed, there are papers that are published that are utter rubbish, some obvious to the point that it is rather surprising that they got through the peer review process at all. I have seen papers published that I knew immediately were wrong or claimed something that didn’t make sense. But, those papers rarely end up having a major influence on the field as a whole: They are usually quickly refuted, or sometimes just kind of ignored and superceded by better work.

Further, a key metric of academic success is publication. As a non-academic I find publication without correctness very troubling.

(1) No metrics are perfect and the metric for academic success is no exception. That said, there are other metrics and, in fact, that is why people have started to look more at metrics such as not just how many papers are published but in what journals and how large an impact in the field have they had (as measured by admittedly imperfect metrics themselves, such as number of citations). I worked in industry for nearly 13 years and the metrics by which people gained success there, particularly in management were surely no better…In fact, I would argue that the correlation between success and merit often seemed to be negative.

(2) As a general point, I find it strange that you find publication without correctness in science to be so troubling. How would science advance if the only science that other scientists learned about at the frontiers of knowledge was science that we were already sure is correct? Nearly all scientists make mistakes and, in fact, scientists are often best known and respected for work that was in large part wrong and superceded by better work. Take the two skeptics, Roy Spencer and John Christy: their most scientifically-respected work is their analysis of the satellite temperature record even though it is now known that their original analysis was plagued by several errors, most of which biased their results in the direction of reducing warming. (In fact, when they first published their analysis, it showed cooling.) It took over a decade of other scientists pointing out these errors and, in some cases, publishing their own independent analyses before there was more general agreement on the results…and, to this day, there remains some areas of disagreement (such as temperature trends in the tropics). Nonetheless, Spencer and Christy are rightfully respected for pioneering the use of the satellite data to study global temperature trends even if they did get it largely wrong.

[Magiver]

@GIGObuster

As mentioned before in another thread, Jones deserves egg in his face for not being good with saving data, however the science is not affected because his research was later confirmed independently.

Other people’s work may hold up but I don’t see how his does. If it’s been confirmed independently then where is “their” data? I’d like to say it ain’t rocket science but it is. I can’t think of a predictive model that would be complicated then weather.

[Lobohan]

@Magiver

Other people’s work may hold up but I don’t see how his does. If it’s been confirmed independently then where is “their” data? I’d like to say it ain’t rocket science but it is. I can’t think of a predictive model that would be complicated then weather.

Climate, not weather. This isn’t about looking at the weather in Peoria on any given Thursday. It’s about finding overall trends.

So the fact that the informed posts in this thread has shown how utterly without merit the OP was doesn’t even make an impact on your opinions?

Also, than.

[Tim_T-Bonham.net]

@Little_Nemo

To put it in perspective, suppose Jones had been studying gravity and submitted reports on his findings. Then he admitted he had never performed any of the epxeriments he had claimed he made. Would this somehow call into question whether or not gravity existed?

Well, it’s only the theory of gravity, after all. Just like evolution.

[GIGObuster]

@Magiver

Other people’s work may hold up but I don’t see how his does.

In the graph shown in the last reply you can compare how it does indeed hold up.

@Magiver

If it’s been confirmed independently then where is “their” data?

As pointed out before, it depends on the deals made with scientific and academic organizations, most of the data is available anyhow.

RealClimate: Data Sources

To look for the data in this specific case the references are here:

http://www.uea.ac.uk/mac/comm/media/press/CRUstatements/guardianstatement

The 1990 paper was only one of a number of papers referred to in the 2007 IPCC Report examining possible urbanizations effects.

References
Jones, P.D., Groisman, P.Ya., Coughlan, M., Plummer, N., Wang, W-C. and Karl, T.R., 1990: Assessment of urbanization effects in time series of surface air temperature over land. Nature 347, 169-172.
Jones, P.D., Lister, D.H. and Li, Q., 2008: Urbanization effects in large-scale temperature records, with an emphasis on China. J. Geophys. Res. 113, D16122, doi:10.1029/2008/JD009916.

Li Q. and W. Li, 2007: Development of the gridded historic temperature dataset over China during recent half century, Acta Meteroloigca Sinica, 65, 293-299 (In Chinese).

@Magiver

I’d like to say it ain’t rocket science but it is.

Sometimes it is, as NASA is involved also.

However, one does not need to be a rocket scientist to realize that assuming that the media will get this subject right is reckless. (The denier media is even less reliable)

@Magiver

I can’t think of a predictive model that would be complicated then weather.

I think you are saying here that climate models can not work with weather, the Met office in England that works closely with the CRU, that was the target of the email hack, develops climate models that also are used in weather predictions.

http://www.metoffice.gov.uk/climatechange/science/projections/

The computer climate models used for the majority of the work at the Met Office Hadley Centre are detailed three-dimensional representations of major components of the climate system. They are mostly run on the Met Office’s supercomputers. As part of the Met Office’s Unified Model, the atmosphere component of the climate model represents the same physical processes as that used for operational weather forecasts. However, due to the longer timescales involved in climate prediction, other components of the climate system are added including:

* Three-dimensional representation of the ocean and sea ice
* An interactive carbon cycle model
* Interactive atmospheric chemistry models
* The coupled atmosphere - ocean - carbon-cycle - chemistry model is known as an earth system model.

For both weather and climate prediction, the Met Office also runs its models at higher resolution over particular regions. Currently regional climate models are typically run at 25 km resolution.

[elucidator]

http://thinkprogress.org/2010/03/02/state-denier-resolutions/

Link to a report in Think Progress, outlining the plans in fifteen different state legislatures to nullify EPA restrictions on greenhouse gases, amongst other “anti-green” resolutions. Prominent amongst the reasons cited is the incredible, dreadful, runaway scandal of Goregate and total debunking of all AGW nonsense.

Jones has been ACORNed. There is also a resolution in Texas to officially rename CO[sup]2[/sup] as “diamond gas”. OK, I made that up.