On the other hand, some people also misuse this truism.
I was reading an article about the well designed study that found that all of the brains of autistic children they studied had the same abnormal prenatal brain growth.
Reading the comments some anti-vaxer dismissed the study by saying “correlation isn’t causation”, and got many pats on the back from other commenters. But unless they can give some reason that parents of babies with this form of abnormal pre-natal brain growth are more likely to give their kids vaccines leading to their autism, I’m going to go out on a limb and say that in this case the correlation is pretty clearly causative.
Will, like so many words was once spelled a variety of ways. Will, woll, wul, wole; these were all options. When people made woll negative, they liked to spell it wonnot which contracts to won’t. We just happened to mix and match a bit when we started to standardize.
Even so its not clear that it is the presence of books that made the difference rather than being raised in the sort of family that likes, and can afford books. It may be that highly educated parents without books, pursued their education for economic reasons but still did not value learning for its own sake as much as those parents who liked books.
Also speaking as a statistician, I know that they may say their models “accounted for” various other co-variates, but in real life non-linear situations it is nearly impossible to fully scrub a covariate’s effect from a given set of data.
In order to really test the hypothesis that it was the books you will need to take a largely bookless population, and give half of them books, and then see if they do markedly better than the other half.
Did they take into account the use of libraries? When I grew up we had some book, not tons, but we had a library which my mother took me to all the time. Her love of books was far more important than the actual presence of owned books - though for families who can afford books, I’d say the correlation makes sense.
I also wonder about the impact of e-books. My younger daughter is in Germany, where it is hard to buy lots of books in English, let alone worry about bringing them back. She does have lots of books on her Kindle, though. I’d guess that books on Kindle should correlate also, if anyone wants to apply for a grant for a follow up study.
Just to be clear, I am not trying to convince everyone that the study should be taken completely at face value. I tend to be as skeptical of studies as the next Doper. I’m just trying to say that so far, the abstract itself seems to refute SlackerInc’s statement. If anyone wants to download the paper, look at the analysis, and report back to the Dope, you would be doing the rest of us a service. Until then, it’s all conjecture.
If critically reviewing scientific research was easy, we’d all have Ph.D’s.
The truth is that it is a skill that must be learned. I had a rigorous science undergrad curriculum, and I didn’t learn it. I was well into my post-doc before I began to readily recognize the common pitfalls scientists make in drawing conclusions.
“Correlation doesn’t equal causation” actually isn’t one of them, IMHO. That’s pretty low hanging fruit. It’s pretty damn hard to get published in a respectable journal if you haven’t accounted for the most obvious co-variates, like the ones the OP or any other layperson would notice. A researcher may not control for them very well (e.g., they may choose the wrong kind of multivariate analysis or model selection method). But at the very least they will acknowledge the weaknesses of their study and insert the appropriate caveats and qualifiers in the discussion. I have never come across a reputable researcher (at least in my field of study) that doesn’t do this. If they don’t do it voluntarily, one of their reviewers will force them to.
No, the problem is that laypeople think the science they read about in The Atlantic Monthly is the same kind of science that scientists are debating behind the scenes. Most scientists don’t sell their work as “surprising new discovery” or “earth-shattering revelation”. That’s what the media does, because the truth is much more BORING. If you actually have a chance to sit down with a researcher and ask them what their research really means, they tend to be quite non-committal, almost overly so. For someone like me who relies on scientists to help shape regulatory policy, it can actually be frustrating sometimes.
You can blame the media if you want, but in my opinion, that also misses the point. It’s not the media’s job to critique the scientific studies they’re reporting on. I wouldn’t trust a science writer, even a competent one, to do a good job of critically reviewing a study anyway. No, the job of the media is to draw attention to exciting research. If the reader is truly interested, they should go to the original white paper and dive into the weeds themselves. And if they are still critical of the findings, then they should contact the primary author (contact info is always provided) and criticize away. But what they shouldn’t do is just automatically assume the research was poorly conducted based on a brief magazine article.
There’s healthy skepticism. Then there’s just anti-intellectualism masquerading as skepticism.
Okay, I was under the impression that you had read the paper. My question was purely out of curiosity. And because I’m very good at coming up with ideas for further research. When I used to sit in at a grad seminar at Stanford the grad students would quiver in their boots as I threw out ideas for five more years of research
But looking at library use would be an interesting thing to do.
Just to be clear, I don’t even think I’m that good at it myself. I usually have to read a paper multiple times before I even feel comfortable summarizing the researcher’s findings, let alone critiquing their methods or conclusions. And I’m not saying that one must have a Ph.D to be able to tear into a paper.
But there is a right way to go about it. People should definitely ask questions like the OP’s. But if they want those questions answered, they’ve got to go directly to the source and not confuse a fluffy magazine piece for actual science, like the OP seems to be doing.
How about next time you criticise a comment, you actually read it closely? I did not refer to education, occupation, or class. I said bookish parents tend to have bookish children.
Say what?!? I targeted the media primarily, and only mentioned academics themselves as an afterthought.
But it also doesn’t mean those who make policy, and those of us who vote for the people who make policy (or appoint policymakers), should not expect to have extraordinary claims proven to us beyond reasonable doubt. (And BTW: “inhaling smoke deep into your lungs all day is bad for your health” is not an extraordinary claim; “getting involved in your kids’ education will just make them worse off” is.) Keep in mind that this is, however inconvenient it can often be, a representative democracy–not an oligarchy run like the old Chinese bureaucracy, by members of the academy. Academics (at least those oriented around public policy rather than private industry) depend on the public (at least the reasonably well-informed subset of the public who follow these issues but do not themselves have doctorates) for funding in many cases, and in any event must convince the public and policymakers to make changes in policy, not just a few members of their own ranks.
Sure, if you (a) think genetics plays no role (something Steven Pinker has pointed out is an all-too-common view among social scientists, while there is no one on his side of the debate that similarly discounts the importance of environment); and (b) you think being “willing” can erase a parent’s lifetime (or even generations of ancestors) of having not accumulated social capital. Count me as extremely dubious.
True! Same holds with things like nutrition and smoking. The cases I’m highlighting are those in which (I’d argue) the most logical, commonsense explanation would be different from what is being promoted.
Right. And one wonders as well whether it can really be a random assortment, like that Frank Sinatra’s character bizarrely owned in the cinematic masterpiece The Manchurian Candidate.
This is ridiculous–you are the one missing the point here. If we are talking about the frontiers of cosmology or the taxonomy of Indonesian beetles, then sure. But this is social science research into educational policy, whose aim is to inform and influence that policy. It’s not just to quietly develop a scientific picture of the world “behind the scenes”. Magazines like The Atlantic are not as erudite as academic journals hardly anyone reads. But within the body politic, they *are *elite. The hoi polloi may read celebrity gossip magazines or maybe even TIME; The Atlantic is a bit more of a select, highbrow audience. And those are the people who ultimately are going to make the greatest impact on whether, and how, educational policy is going to change.
To expect such people to go and “dive into the weeds” for every research result that comes down the pike, while they continue to work at their own occupations and raise their families, is just beyond reason. It is reminiscent of the insanely complex Jim Crow-era literacy tests, which guaranteed the right of blacks to vote in theory but scuttled them in practice.
And even leaving the political policymaking aspect aside: this writer is flat out telling parents, in an influential publication (as well as on podcasts and so on), “don’t help your kids with homework or get involved in their schools: you’ll just make everything worse”. So even if individual parents just take this advice in their childrearing, it could destroy many children’s lives. But I’m not supposed to be critical of the media?!?
Are you calling me an anti-intellectual? Hah, that’s a new one! I’ve been accused of being an elitist, a moniker I don’t entirely disavow; but never an anti-intellectual.
Policy makers are not dictated by The Atlantic. They either have a staff of professionals whose job it is to know the research, or they hire consultants who do. At least in government, almost every policy can be traced back to technical document(s), which are drafted by scientific and technical committees, who are comprised of researchers or those who are heavily steeped in research. Those technical documents do not cite publications like The Atlantic. They cite journal articles.
What individual people choose to do with the information they pick up from the environment is not the same as institutional policy.
Um…that comparison is just so over-the-top crazy that I’m really not taking you seriously right now.
Most people reading a magazine article aren’t going to remember it an hour later. Maybe they’ll mention it in passing at the water cooler the next day if something triggers their memory. And that’s it. For people who are actually interested in the content, whether for professional reasons or because of a personal passion, they would do well to follow-up on their own. They will probably go to the source or to a publication that treats the study more critically. As they should.
I never said you can’t be critical of the media. I haven’t read the article or listened to the podcasts, but if she is making these kinds of statements, without the appropriate qualifiers or hedging, then I agree with you that she’s being irresponsible.
I just think the media has a different standard than science does. I’m not going to expect a science reporter to ask hard-hitting questions UNLESS the scientific discourse has reached such a high-pitch to warrant that kind of attention.
If you want to infer from my post that I’m calling you an anti-intellectual, go right ahead. But I didn’t call you anything. I don’t know you or your posts well enough to call you anything. Yet.
But from what I’ve seen, the people who react to these kind of studies act like the media is accurately reflecting the science - and it was those people who I was referring to. You’ve seen it - the “big time scientist spends money proving something we normal people already know.” Or, like in this case, some think the scientists are too dumb to realize that correlation is not causation. The media only has space to cover the big story, not the provisos and quid pro quos.
Politicians - either out of ideology or the desire to grandstand and show the voters that they are just like them or out of ignorance are prime culprits - Proxmire being the most famous example.
Science writers have a very tough job of digging into a vast variety of fields, trying to understand who is on which side of the controversy, and understanding it well enough to simplify for the masses without over simplifying it. When I was involved in selling a very complex piece of engineering software I got interviewed by people from the trade press, and I was quite impressed by the broad if not very deep knowledge they had of a diversity of subject matter. So even the Atlantic is going to get it wrong or not perfect. We their readers have to give them and the people they interview a break.
Again, we don’t have that type of technocratic government. Look at what’s happening with Common Core. The general public, or at least the moderately well informed subset thereof, needs to buy in. But in any event, the choice of whether to get more or less involved with one’s children’s schools rests with indivudual parents, not with policymakers.
I am going to go out on a limb and guess you are not a parent? Most parents I have ever known treat recommendations like this much more seriously than that. Again, we are not talking about pure science that is aimed, for laypeople, at eliciting a “huh, interesting, a new subatomic particle”. We are talking about the most fraught subject I can think of: How Not to Screw Up Your Kids.
A point of agreement, hey.
Weird how people keep saying things like this, implying I’m a newbie. I have been commenting on the SDMB for many years (even going back to the AOL days, when I was AlanKngsly).
In theory, at least, yes we do. Not every reg is backed by science, but policy makers at least pay lip service to it because it makes whatever they dictate easier for the stakeholders to swallow. And the way they pay lip service to it is by forming independent technical advisory committees/panels, who are tasked with bringing science to institutional practices. Stakeholders (the folks who are most affected by arbitrary and capricious policies) demand this kind of rigor and are quite willing to sue if they think it’s not happening.
I’m not saying government never has hare-brained schemes. But department/agencies do have institutionalized requirements that force beaurocrats to interact with egg-heads, who will then hopefully dissuade them of any crazy notions they may gotten from NPR or their bathroom magazines.
Sorry, I don’t have a good memory for names. With a few exceptions. all of you people are the same to me.
On that note, it’s an irony of history that one of the big proponents of the ‘tobacco does not cause cancer’ thesis was R. A. Fisher- the guy who discovered a good deal of modern genetics, statistics, behavioural ecology and evolutionary biology, and who is one of the most influential scientists of the last century. Fisher was also an avid smoker, and apparently was extremely indignant at the idea his hobby was bad for you.
Sometimes even incredibly smart people can make mistakes.
Here’s another one, from ten days ago. This time we hear from the horse’s mouth: the Harvard researcher who published the study. In this case, the explanation offered strikes me as idealistic wishful thinking, almost the opposite of what I suspect is the true reason for the pattern discovered. Basically, the hypothesis is that scientific papers written by a multicultural group of researchers are “better” (because more cited) than those written by a more monocultural group:
As I was listening to this, I was already musing to myself “or is it because people are more likely to cite a paper that has an author of their own ethnicity–therefore, if you ‘cover your bases’ you’ve got a little hook for everyone?” And then almost comically, another point made later in the story would, one would think, seem to really underline this possibility–yet it never seemed to dawn on anyone, sheesh:
So he noted that people tend to like to work in labs with people of their own ethnicity, but it never occurred to them that they might like to cite *authors of their own ethnicity, irrespective of how “good” the papers are? Good grief.
*ETA: Or, for that matter, of their own regional affiliation.
I don’t know if they looked at geographical diversity, or just, for instance, different ethnicities within the same university, but another factor is that truly interesting, important papers are likely to require a wider range of expertise than more routine papers. So if someone in the UK is collaborating with someone in China, the odds are good that they’re looking at something in a new way. If it’s more routine, it’s more likely that some guy down the hall can do the needed job.
Also, just in general, there’s likely to be a correlation between the importance of a paper and simply the total number of authors in general. A paper that required ten authors is more likely - on average (obviously there are zillions of exceptions) - to be more interesting than one with two or three authors. At the very least, it’s going to have more data in it. Obviously, the more authors there are, the more chance there is to pick up an ethnically diverse mix.
The report in question may have considered and accounted for these effects. I haven’t read it.
