I ask because of this. One of the pillars of scientific credibility is the ability to reproduce results. If you can’t do that how seriously should the conclusions of the study be taken?
A huge amount of money and effort is spent addressing psychological health, often based on assumptions derived from the results of studies as to this or that approach being the most effective. If it’s really just a crapshoot are we squandering resources in the wrong directions?
I’ve thought for a while that academic publishing is in a crisis and the traditional academic journal as we know it is in a process of long, slow decline. I don’t know what will replace it in the long term. The problems are numerous. There are too many journals, and too many of them of low quality. There’s too much pressure on academics to publish articles in quantity rather than quality. The prevalence of fraud. And hoaxes. More and more people are becoming aware of the problems, and are starting to wonder how things went so wrong.
Psychology hardly qualifies as a field of science to begin with. Much of psychological theorizing is barely above the level of mysticism and invokes so many abstractions to the actual mechanics of cognition that it is one of those areas of research where the conclusions are often so useless that they are “not even wrong”. Unfortunately, we really lack enough knowledge of how the brain produces cognative behavior to be in any way useful at treating psychological disorders outside of trauma, tumors, and certain gross neurochemical imbalances, whereas various forms of talk therapy can produce positive results, although it isn’t clear if they have greater efficacy than just talking to close friends, confidants, et cetera.
There is, of course, the problem of researcher bias; of course a researcher wants the study to validate his or her hypothesis, and unlike the hard sciences where the data may be collected and assessed objectively, psychological behavior is always interpreted by the observers who filter it through their own perceptions. So getting false positive results doesn’t require perfidy or avarice, just a normal human tendancy to err toward a favorable interpretation.
Stranger
Publication bias and regression to the mean happens in all fields. Remember hormone replacement therapy for women? The first studies were all “wow it’s great” and then as more studies were conducted, the actual truth started to become apparent. That happens in every field on every topic-the first published studies are overwhelmingly positive and then later studies tend to not be so positive or even negative. Which is why people look at systematic reviews and meta-analyses in order to figure out what is more likely to be true instead of looking at single studies.
Although I do have to agree that psychology doesn’t really qualify as a science.
Psychology has never been subjected to scientifically verifiable tests, and cannot be under current humanitariian paradigms. There is a rigorous social objection to doing any real psychological study, because doing so would require human laboratory subjects placed in control groups not consistent with free will and liberty, not to mention dignity. You can’t treat babies the way a metallurgist treats iron in order to reach scientifically valid conclusions. That social restriction will forever be a wall that psychology, sociology, pedagogy, and all the rest of human behavioral sciences will never be able to penetrate in order to reach scientific rigor…
We should probably differentiate between serious psychological studies published in academic texts and the studies that end up as sound bites on talk shows. The former may be unreliable but the latter usually are. And the sound bites are the ones everyone can quote.
“Everybody knows” that homophobes are turned on by gay pornography. Studies “prove” it. Nobody knows who did the studies or whether the studies are reliable, but who cares? That’s now a “scientific” fact, and you wouldn’t want to be one of those science deniers, would you?
The studies that cannot be confirmed or replicated *are * “serious” published studies.
They expected 50% and got 36? That doesn’t seem all that bad.
The first problem with science is that it takes a while for the process to churn out decent results. The second problem is that you don’t really know whether the churning is done or not at any given point.
And that’s science in general where 50% of papers are basically incorrect. Psychology is much worse because of its largely subjective nature, the difficulty of doing research on people (although medicine doesn’t use that as an excuse) and last but not least, because incompetence and fraud run rampant in the field. So 36% isn’t nearly as bad as I’d expected but yes, if something important depends on it, ignore the psychology studies because they’re more likely to be incorrect than correct.
What on earth are you talking about? Nobody commissions a study in order to cite it on a talk show.
You know damn well what I’m talking about. The public generally knows little or nothing about serious studies conducted by reputable psychologists and published in scholarly journals. But the worthless studies are the ones that end up cited in popular mass media outlets and quoted on internetvcgat rooms.
I agree with astorian. Scientists generally toil in the unknown, yet make huge strides with some of the things they study.
However, publish one interesting or crazy thing (Scientists discover having sex more often leads to smarter children!) and you get on Buzzfeed and CNN’s “look at this funny story” segment.
Now you have a name, now you’re “famous”, and now you can have the attention on money to do other work…real or not.
It would be a really risky game to play, but a scientist could fight for funding on something they really care about. So they do a media-friendly study (Cats are the key to stronger friendships?!?!) and get enough recognition for their school/lab to do the real work.
IANAP but I’m not sure this is fair if you’re talking about modern psychological researchers (as opposed to, say, Freud and his disciples), who at least try to follow the scientific method.
We should “believe them” (not really how science works) based on their individual scientific merit, as with all scientific studies. And that is aided by referencing the actual journal article in question (10.1126/science.aac4716), not (or at least in addition to) some summary by a journalist.
Of course, that will not stop those who know little about the topic from opining about it.
Of the 100 studies in question, are there any particular ones that stand out to you as “barely above the level of mysticism” or “so useless that they are ‘not even wrong’”?
I haven’t got a fucking clue what you’re talking about. Can you give me an example of one of these “studies” written for the popular press?
I once took a senior level industrial and organizational psychology course as part of my undergrad coursework.
It was kind of eye-opening in that psychological studies are heavily statistically oriented and seemed to be centered around trying to correlate observed behaviors with models that explain them.
And the studies often contradicted each other to some degree, and the correlations were weak.
While I realize a lot of that is a consequence of not being able to rigorously experiment in an ethical fashion, it did throw a lot of personal skepticism in there as far as what psychologists actually know. I mean, our prof wouldn’t say anything absolute like “Your average soldier is only good for around 200 days of combat in total.” Instead, they’d always hedge and prevaricate with stuff like “Moran’s 1966 study shows that soldiers are good for 200 days of combat under conditions X, Y and Z, while the Appel study during WWII found that soldiers are good for 200-240 days of combat.”
It seemed very much to me like it was very loosey-goosey and indistinct compared to something like the math, physics, geology and even biology that I had already been taking as my science/math credits.
Well, yeah. That’s what happens when you can’t create a true control group.
I think the real question is “What are the alternatives?”
We need ways to understand behavior. We need ways to treat mental illness. We might argue about whether Higgs bosons matter, but psychology is right here and now by any definition.
So we study it in the best way we can… and if studies of our studies show that we don’t do studies very well, I think the response is to study a better way of studying. We certainly can’t stop studying it, though.
(If the word study has lost all meaning for you too now, raise your hand and someone can write a paper about it. 
)
A reasonable argument if the studies lead to something positive but I don’t think they do. The same argument could be used to excuse Frued and his methods but they are largely regarded as bunk. How much effort was wasted interpreting dreams or trying to prevent a child from being anal retentive?
At this point I’ve decided that psychological studies are mostly useless until further notice.
I can agree with that viewpoint on the issue, but is there a better guide?
If you’re a parent raising a child, you have to make decisions like whether to spank or not to spank and you have to do that based on whatever evidence is around. People like Freud are clearly full of crap, but you could do a lot worse.
I haven’t read all of Freud, but what I have read had no statistics and no mention of MatLab. Modern psychology papers based on studies do.