# Studies do not say anything **URL:** \ **Category:** Great Debates\ **Created:** [May 26, 2024, 1:52am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080 "2024-05-26T01:52:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [May 26, 2024, 1:52am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/1 "2024-05-26T01:52:07Z")
Nothing in life is perfect. Usually, the imperfection is that people don’t check what science is saying. However, there is also sometimes the lesser known issue occurring of people checking what the science is saying, so let me give an example. > [@](#): > > **Hypothesis:** > > God exists. > > **Methodology:** > > Researchers traveled to a location known as “God’s Spot”. We placed a paper bag, opening down, above the known location of God’s Spot and observed the paper bag lifting off the ground on its own and taking flight for tens of seconds. > > **Conclusions:** > > God made the paper bag fly. God exists. Or, you know, there’s a naturally occurring source of lighter than air gas venting up through the sand in that spot. (Which is plausibly God’s doing, but so is everything else if he made everything, so that’s not more evidence than chronicling any other part of our existence.) Ultimately, the only part of a paper that’s worth a damn is the middle part that gives the actual details of what was done and what the results were. The other parts are, in some sense, just an opinion piece on those results and their meaning. (Not to say that a knowledgeable person’s opinion is without value - they may be laying down the solid truth and they may be able to provide context that helps to understand the experiment and its results better. But there is no law of the universe that this is so. Some research scientist in the world passed with all D’s and anyone on the internet can write and submit some paper that’s in the correct format to somewhere and have it look official.) But now that we’re only giving any true weight to the middle part, we still have to accept that it only tells us what it tells us. Paper bags float off when placed opening down at God’s Spot. It demonstrates nothing more and nothing less than that. [https://www.reddit.com/r/xkcd/comments/1apvjzl/any\_other\_xkcds\_that\_make\_showing\_or\_applying/](https://www.reddit.com/r/xkcd/comments/1apvjzl/any_other_xkcds_that_make_showing_or_applying/) How much value to take from the research sort of depends on **your ability** to grade the quality of that research. You may well not be qualified to do that, and nor might be the average science journalist who is reporting on it, nor the average social media influencer, and they’re liable to take the author’s more user friendly summation as gospel, without even reading the methodology. So you might say that the best strategy is to find an expert in the field to listen to since they have the chops (presumably) to evaluate this stuff? No, we know that doesn’t work since there are any number of scam influencers with the qualifications to know better and how are you to know which to back, or even if you should pay attention to either one? [![](https://img.youtube.com/vi/S718YtbwrzY/maxresdefault.jpg "Doctor Blocks Me For Correcting His Dangerous Advice") ](https://www.youtube.com/watch?v=S718YtbwrzY) [![](https://img.youtube.com/vi/4xuxFyS0Hls/maxresdefault.jpg "Exercise Scientist Critiques Dr. Joel Seedman's Absurd Training Antics") ](https://www.youtube.com/watch?v=4xuxFyS0Hls) [![](https://img.youtube.com/vi/yEsPqB2bI6A/maxresdefault.jpg "Dr. Eric Berg Gets Fact-Checked By Medical Doctor") ](https://www.youtube.com/watch?v=yEsPqB2bI6A) [![](https://img.youtube.com/vi/z1cOvH1_vKc/maxresdefault.jpg "Dr.Berg Exposes Dr.OZ Show") ](https://www.youtube.com/watch?v=z1cOvH1_vKc) I’ve generally suggested to people that they should follow the guidance of large organizations of experts, like the American Heart Association or the American Physical Society since anything they promote has to have been signed off on by a committee. The evidence has to have satisfied a majority of the experts. But, even there, we know from history that sometimes the off-the-wall, long shot theory that “all the experts” poo-poo’ed on, ended up becoming the accepted answer. Some branches of science - for example, psychology and macroeconomics - may be relatively far behind some other areas of academics (but catching up). Still, probably the best answer is to trust associations of experts. Past that, actually read the research, mostly read the middle part, try to learn statistics and how to lie with statistics, try to be conversant in the basics of all things, but most importantly accept: All things that things you’ve ever heard or said are just some idea some human had. That idea might eventually prove to be wrong. ---
**Author:** ![Jackmannii](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/jackmannii/32/311_2.png) [@Jackmannii](https://boards.straightdope.com/u/Jackmannii)\ **Post date:** [May 26, 2024, 2:52am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/2 "2024-05-26T02:52:09Z")
> [@Sage\_Rat](#): > > I’ve generally suggested to people that they should follow the guidance of large organizations of experts, like the American Heart Association or the American Physical Society since anything they promote has to have been signed off on by a committee. The evidence has to have satisfied a majority of the experts. > > But, even there, we know from history that sometimes the off-the-wall, long shot theory that “all the experts” poo-poo’ed on, ended up becoming the accepted answer. Here’s something I saw [over at Respectful Insolence](http://respectfulinsolence.com/2024/05/24/antivax-tech-bro-steve-kirsch-reinvents-challenging-a-scientific-consensus/) today that seems relevant. > [@](#): > > Because a scientific consensus is established by many studies that usually include lines of evidence from different scientists and scientific disciplines that converge on a single conclusion, which becomes the scientific consensus, it generally requires a lot more than one experiment or study to overthrow it. While it is true that, in theory at least, a scientific consensus can be overthrown by a single study, that rarely happens, if only because a finding that is so anomalous compared to existing evidence is usually far more likely to be erroneous than it is to be correct, which is why scientists will demand replication before accepting it. > > Here’s how a scientific (consensus) is usually modified or rejected. It often starts with a single study (or group of studies) that cast doubt upon the existing consensus and lead to more studies that also cast doubt upon the consensus. Ultimately, the accumulation of new data and new experiments will end up leading scientists either (1) to modify the old consensus to account for the new evidence or (2) less commonly, to reject it in favor of a new consensus. It’s freakishly rare for a bolt of scientific lightning from out of nowhere to overturn a large body of painstakingly accumulated and diligently replicated research. The chances of Some Guy On Social Media Doing His Own Research and overthrowing rigorous scientific consensus is zero. There are many areas in medicine and science where I don’t have the qualifications to understand exactly how research was carried out and don’t have the skills/chops/time to evaluate that “middle part” of publications, including statistical analyses. That’s why to a large extent I depend on the qualifications and skills of experts in a particular field, to put papers in proper perspective. The arrogance of so many people these days who resent and belittle experts is hard to believe, but that’s where we are. ---
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [May 26, 2024, 4:23am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/3 "2024-05-26T04:23:05Z")
> [@Jackmannii](#): > > The arrogance of so many people these days who resent and belittle experts is hard to believe, but that’s where we are. Well, so, that’s where the debate begins. First, I would submit that there is research that’s poor, that doesn’t mean what the researcher believes it to mean, and that people shouldn’t treat scientists and researchers as infallible gods. > [@Hydroxychloroquine study "does not meet the Society's expected standard."](https://boards.straightdope.com/t/hydroxychloroquine-study-does-not-meet-the-societys-expected-standard/851443/4): > > There is an argument that good science sets the parameters in advance and follows through with it. Fiddling with your methodology after the fact is one way to create results out of nothing, because you can find the things that was, by random happenstance, an outlier variable. Rolain’s methodology was to perform mouth swabs. He ignored people who he couldn’t swab - like people who died. That he stuck to it is, in the previous sense, good. But it is, obviously, a bad methodology. Sticking to a … As you say, that’s me in that thread - not a doctor - critiquing the research output of a doctor. Likewise, here I am complaining about the IHME prediction model of COVID fatalities (which I believed failed to incorporate lag times between case counts and fatality counts, and to assume a bell curve rather than a fast rise, slow fall curve that epidemiology models usually predict). > **[IHME Prediction](https://publius2point0.wordpress.com/2020/04/22/ihme-prediction/)** > > Patients come to a hospital and, after one week, either they go home cured (50% of the time) or they die (50% of the time). On day 1, 1 new patient arrives; on day 2, 2 new patients arrive; on day … And in other threads on other topics, I have debated the merits of research, doubting the quality of their methodology and speculating on confounders, spurious correlations, etc. How do I know that I’m qualified to do so? Am I always wrong to do this, minus an applicable degree? Can I say that public debate, hopefully populated with experts who can respond to questions and concerns, is a positive activity? Or does that spread doubt and the willingness to doubt science among those who probably shouldn’t do so? Is there any safe way to choose individual experts to follow? Can I trust Gil Carvalho? Where are the boundaries of taking in and communicating over raw research? ---
**Author:** ![Pitchmeister](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pitchmeister/32/3459_2.png) [@Pitchmeister](https://boards.straightdope.com/u/Pitchmeister)\ **Post date:** [May 26, 2024, 8:31am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/4 "2024-05-26T08:31:30Z")
> [@Sage\_Rat](#): > > Where are the boundaries of taking in and communicating over raw research? Covid completely revolutionized the public interest in and public access to scientific research in its raw form. In some ways, that was a good thing, as it gave much needed information into the hands of people who needed it. In other ways, it gave some people the impression that scientists are just guessing, when a particular study turned out to be wrong upon peer review. It used to be that only absolute experts in a field would read unreviewed studies off a pre-print server, so the large public only got the filtered version you would read in a newspaper a year later. This situation has apparently encouraged many people to consider themselves scientists, when they are not equipped to judge the merits of any particular methodology. I believe it’s probably a good thing to „democratize“ access to scientific research, so everyone can be informed, but people should absolutely know their limitations. ---
**Author:** ![glee](https://avatars.discourse-cdn.com/v4/letter/g/b5a626/32.png) [@glee](https://boards.straightdope.com/u/glee)\ **Post date:** [May 26, 2024, 9:29am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/5 "2024-05-26T09:29:41Z")
Sadly some politicians and media groups simply lie without any backing evidence, which is far worse than your point. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [May 26, 2024, 9:43am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/6 "2024-05-26T09:43:15Z")
> **[Significant](https://xkcd.com/882/)** > > 'So, uh, we did the green study again and got no link. It was probably a--' 'RESEARCH CONFLICTED ON GREEN JELLY BEAN/ACNE LINK; MORE STUDY RECOMMENDED!' ---
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [May 26, 2024, 3:32pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/7 "2024-05-26T15:32:16Z")
> [@glee](#): > > Sadly some politicians and media groups simply lie without any backing evidence, which is far worse than your point. But your average person often fails to catch it because they are paying attention to nonsense fact spreaders. If the “experts” are saying that vaccines kill and Mexico is subsidizing Islamic terrorism into the nation then you need to hire the politician who will act on these things. That’s, on the face, pretty rational. Right now, we have some politicians saying that we need more “Shoot first, ask questions later” options on the border, while the elected labor group of the border patrol, the leader of whom endorsed Trump in 2016, is saying that the new border legislation would be sufficient for our needs if the Freedom Caucus would pass it. If people were paying attention to the expert group, Rather than individuals, we’d be in a different position. ---
**Author:** ![Max\_S](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@Max\_S](https://boards.straightdope.com/u/Max_S)\ **Post date:** [May 26, 2024, 3:56pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/8 "2024-05-26T15:56:56Z")
I think the primary role of experts is to explain, not dictate. But this is just my opinion. Not everyone thinks the ideal society is an enlightened democracy. Traditional systems such as Confucianism, to the best of my understanding, promote a social heirarchy where people stay in their lane. Leave the science to the scientists and the farming to the farmers. Catholicism is somewhat similar, depending on who you ask. Compare also Plato’s _Republic_. ~Max ---
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [May 26, 2024, 4:06pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/9 "2024-05-26T16:06:07Z")
I don’t think that Confucianism holds up to scrutiny but I feel like that would go too far afield of the central topic, so I’ll just leave that at that. ---
**Author:** ![Max\_S](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@Max\_S](https://boards.straightdope.com/u/Max_S)\ **Post date:** [May 26, 2024, 4:56pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/10 "2024-05-26T16:56:20Z")
> [@Sage\_Rat](#): > > **Hypothesis:** > > God exists. > > **Methodology:** > > Researchers traveled to a location known as “God’s Spot”. We placed a paper bag, opening down, above the known location of God’s Spot and observed the paper bag lifting off the ground on its own and taking flight for tens of seconds. > > **Conclusions:** > > God made the paper bag fly. God exists. Let me be more explicit. This is a straw man of the scientific method. Between methodology and the conclusion come the most important sections, **results** and **analysis**. You’ve botched it by putting results in methodology and leaving out analysis entirely. Even the hypothesis is flawed. I’m giving you an F on this project, Mr. **Sage\_Rat**. Scientists are experts not only in coming up with experiments (methodology section) and reporting conclusions, but also in _explaining facts and conclusions_. The primary purpose of a study to the general public is not, “here’s a cool idea for an experiment”. And it is not to tell the public what to think. It is to report facts and explain how they lead to conclusions. ~Max ---
**Author:** ![FlikTheBlue](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/fliktheblue/32/10972_2.png) [@FlikTheBlue](https://boards.straightdope.com/u/FlikTheBlue)\ **Post date:** [May 26, 2024, 5:01pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/11 "2024-05-26T17:01:03Z")
> [@Sage\_Rat](#): > > I don’t think that Confucianism holds up to scrutiny but I feel like that would go too far afield of the central topic, so I’ll just leave that at that. IMHO it’ a good example of two ways of trying to understand the world. There’s the authoritarian “believe what I say because of who I am” vs. the authoritative “believe what I say because the evidence says this is the best explanation we currently have, and here’s the evidence”. Confucianism is just one example of the authoritarian model of seeking knowledge. Part of what’s going on is that authoritarians present the greatest advantage of the scientific method, the willingness to throw out an old explanation when new evidence shows it to be incorrect, as a flaw. ---
**Author:** ![Max\_S](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@Max\_S](https://boards.straightdope.com/u/Max_S)\ **Post date:** [May 26, 2024, 5:25pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/12 "2024-05-26T17:25:33Z")
I wouldn’t describe Confucianism as authoritarian in that sense. The gist is not that “experts are infallible” so much as “don’t be a backseat driver”. Like, if you aren’t in the field of medicine, maybe you shouldn’t be reading medical journals with an eye to challenge the research in the first place. But there’s a reciprocal obligation of the medical establishment to do right by the rest of society. ~Max ---
**Author:** ![Exapno\_Mapcase](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/exapno_mapcase/32/1051_2.png) [@Exapno\_Mapcase](https://boards.straightdope.com/u/Exapno_Mapcase)\ **Post date:** [May 26, 2024, 6:54pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/13 "2024-05-26T18:54:11Z")
Adult-onset lactose intolerance (LI) took until about 1970 to be widely recognized as a real thing. Once that happened, teams of researchers went off around the world measuring the percentage of LI in countries and tribal peoples (the term used then), as well as doing experiments on undergraduates. A flood of papers were published in every medical journal, along with a variety of anthropology journal articles tracing the origin and spread. Almost none of this made it to the general public, except for one _Scientific American_ article. Lactaid pills didn’t appear until 1984, the milk later. Lucky me, I got diagnosed in 1978. I had to do my own research, before the internet. In 1983, I decided to write the first book on the subject. The basement of my local School of Medicine was filled with old bound copies of every major medical journal world in the world. I started reading. A year later I had read every article on LI ever written, as well as all the anthropological literature in another school library. I may have been the only person in the world to do so. And at least 50% of it was sheer junk. Those wandering researchers used different tests prepared in different ways on non-randomly selected populations creating results that could not legitimately be compared. The statistics used were so basic that I could judge them and find them wanting. An incredibly tiny proportion of the studies contained information that would be of any use to the ordinary consumer. Almost all of them were written for no better reason than to gain a credit for their CV. Humorously, when genetic studies finally rolled around several decades later, the geneticists roved around the world in the footsteps of the anthropologists and came up with the same conclusions about the spread of LI - without any credit to the studies I had read thirty years earlier. You might say that I became jaundiced as to the value of the average peer-reviewed printed in prestige journal article. Unquestionably, other articles on other studies in other fields are life-saving. And unquestionably, only an expert extremely well read in that field will be able to spot them. ---
**Author:** ![Max\_S](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@Max\_S](https://boards.straightdope.com/u/Max_S)\ **Post date:** [May 26, 2024, 7:03pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/14 "2024-05-26T19:03:26Z")
Interesting. Did you find a correlation in the quality of a study and how frequently it was cited with approval? Because that’s tracked pretty well in the modern day and it’s a good indicator of consensus in my experience. Or in other words, did you find a consensus in 1984 and was it quality science? ~Max ---
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [May 26, 2024, 8:50pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/15 "2024-05-26T20:50:21Z")
> [@Max\_S](#): > > Scientists are experts not only in coming up with experiments (methodology section) and reporting conclusions, but also in _explaining facts and conclusions_. In theory, that’s true. But there’s that thing about theory and practice. (You forgot Disclosures - probably the most important section other than the middle.) > [@Sage\_Rat](#): > > (Not to say that a knowledgeable person’s opinion is without value - they may be laying down the solid truth and they may be able to provide context that helps to understand the experiment and its results better. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [May 26, 2024, 9:05pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/16 "2024-05-26T21:05:35Z")
> [@Sage\_Rat](#): > > (You forgot Disclosures - probably the most important section other than the middle.) Well, it can be. But there are a heck of a lot of studies where there’s simply nothing to disclose. ---
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [May 26, 2024, 9:13pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/17 "2024-05-26T21:13:02Z")
The ones most likely to end up out in the wild are probably the ones most likely to have a disclosure hidden down at the bottom. Maybe it’s not particularly relevant in sociology but for research into supplements, hair tonics, beauty procedures, etc. - the sort of thing that a normie might look up - they’re pretty rampant and good to pay attention to, I’d suspect. Or for something like border patrol, knowing who you’re reading and what biases they might have - generating your own mental “disclosures” - is also pretty good to do. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [May 26, 2024, 9:33pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/18 "2024-05-26T21:33:35Z")
Or the effectiveness of hydroxychloroquinine for covid. If a researcher is affiliated with a company that sells hydroxychloroquinine, then that’s definitely a relevant disclosure. If they’re affiliated with a company that sells other anti-covid medications, that might be. But if they’re not affiliated with any drug company at all, what are they supposed to disclose? And of course, the real problem isn’t when people don’t read the disclosures. The real problem is when a study should have disclosures, but doesn’t. How does a layman tell the difference between a study that doesn’t have any disclosures because there’s nothing to say, and a study that doesn’t have any disclosures because the researchers unethically left it out? ---
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [May 26, 2024, 9:43pm UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/19 "2024-05-26T21:43:43Z")
> [@Chronos](#): > > The real problem is when a study should have Sure, but 100% of people who declare a conflict of interest have a conflict of interest. The rest are a lower percentage. ---
**Author:** ![Exapno\_Mapcase](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/exapno_mapcase/32/1051_2.png) [@Exapno\_Mapcase](https://boards.straightdope.com/u/Exapno_Mapcase)\ **Post date:** [May 27, 2024, 12:41am UTC](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080/20 "2024-05-27T00:41:05Z")
Databases of frequency of citation may have existed in 1984 but I never saw one. [Next page](https://boards.straightdope.com/t/studies-do-not-say-anything/1002080.md?page=2)