This would be an interesting thing to investigate but they misclassified one statement in their own study invalidating their own conclusion. I’ve spoilered the offending question in case you want to look at the survey first.
[spoiler]They classified the statement “ISIS lost a significant portion of its territory in Iraq and Syria” as factual.
Significance is an opinion. If ISIS lost 100 square meters I might consider it significant if my house was in there or if it was ISIS’s last territory. If ISIS lost a 100 square kilometers, it might not be significant if it was all empty desert or if they continued to hold 1 million square kilometers. In any event, all these determinations about significance are just my opinion. Even “statistical significance” is merely a consensus opinion on what we consider to be worthwhile evidence.[/spoiler] This misclassification calls Pew’s results into question. I don’t see how this study saw the light of day or how it was uncritically written up by the press. In a study that could provide some insight into how people understand the distinction between news and commentary, Pew failed to show that they know the difference. Anyone think differently?
I almost want to believe that Pew is conducting a meta-study on how the press will report on what is very obviously junk science without questioning the results. Perhaps Pew will publish the results of that study in the future.
I believe there was another study done some years ago where some psychologists were testing if political affiliation affected belief in factual vs. non-factual statements that made the opposite error. One of the questions was “No WMDs were ever found in Iraq”, and they said it was factual. It isn’t - some WMDs were found in Iraq. What I think they meant to say was “No significant amount of WMDs were found in Iraq” and, as you say, “significant” is opinion.
I bet neither Pew nor the psychologists were even aware of what they were doing, which is an interesting artifact on its own account.
I think T&C is technically correct about the “misclassified” question, but it doesn’t strike me as invalidating the survey.
It’d be nice if Pew did a survey lifting “news” items out of print and electronic media and asked the public (or journalists) which of them were actually news and which were opinion presented as news.
It would also be fun to do a similar survey of Dopers regarding the tendency to render opinions in the guise of news (typical response: “Well, it’'s twue!”).
In science, significance is not really an opinion, but just saying it was above the margin of error.
I’m not saying this applies here, but it might be why significance was not thought of as being an opinion word.
We had these tests back in math class of all places, since we needed to understand logic to do geometry.
I don’t agree that a single option invalidates results, however. It means that you may need to recalculate with that question thrown out, but that data should be available to do that.
I didn’t catch that one as “misclassified” when I took the survey, but then when I read the spoiler box, I could see Tired and Cranky’s point.
But I think it’s a borderline case. If the statement had been worded “The portion of its territory that ISIS lost in Iraq and Syria was significant,” T&C would have a stronger case that it should be classified as opinion, since the emphasis would be on whether the amount of territory lost qualifies as “significant.” But the way the statement actually was worded, the emphasis was on whether ISIS had lost territory, which is a matter of fact.
I think Shodan understood that. I believe that he’s referring to a different study where Pew was evaluating whether the public understood certain facts but they themselves got one of their “facts” wrong. I vaguely remember this study too.
Again, statistical significance is a consensus opinion on what is good evidence. If I do a study that says with 75% confidence that something is true, my opinion might be that my test is statistically significant. Others in the scientific community who have reached a consensus that 90% or 95% confidence is the minimum statistical significance would disagree with my opinion. These are both still opinions.
It’s also true that nothing in Pew’s disputed statement suggests they are using “significance” in a scientific sense. They are using it qualitatively as a synonym for “important,” which I hope we can all agree is an opinion.
I do agree that Pew could rerun the analysis throwing out responses to the bad question. Until they have done so, in my opinion, their conclusions are invalid.
In the study I highlighted, Pew was trying to gauge whether people could distinguish between an assertion of a fact (whether or not it was true) or an assertion of an opinion (which is not falsifiable).
When I was taking the quiz, I didn’t worry about whether the statements were true or not, just whether they were the kind of statements that would be classified as Fact vs Opinion. So, for example, if one of the statements had been “Los Angeles is the capital of California,” I would have classified it as “fact” (as opposed to “opinion”) even though it’s a false “fact.”
Yes. The way the statement is presented should tell you whether there is a possible factual response.
A pound has 32 ounces. This is a factual statement, it is wrong, but a factual statement. It can be measured and found to be true or false, but the statement is a factual one.
Chocolate cake is the best tasting cake. This is an opinion. There cannot be a factual response even if you believe that it is true. It isn’t measurable.
Pot should be legal in Texas. This is an opinion.
Pot is legal in Texas. This is a factual statement. It’s wrong but is statement rather than opinion. Because it can be measured, checked, yes/no.
Because one question in ten was ever-so-slightly ambiguous, you label the entire study as “junk science?” Yes, they should have been more careful about using a specific measure instead of “significant”, but even if you toss that statement into the “opinion” camp, the data from the other nine remains relevant.
ETA: Perhaps more interesting is that (for an easy test), about 70% of people disagree with you (i.e. are shown as answering “correctly”), more or less in line with the other questions, so it looks like in general folks were treating “significant” as though it were a placeholder for a hard number.
What they mean by “factual” is a verifiable statement. Empirical data exists about the territory controlled by ISIS, which puts the statement in the category of factual, regardless of whether one agrees that the change is significant.
I got all questions correct. The ISIS one immediately hit me as a factual statement and not just because I’m accustomed to treating “significant” as an measurable property. It’s factual to me because the main point being communicated (that ISIS lost territory) is not a subjective assertion or value judgement.
No, the statement is about the significance. The statement is equivalent to “the amount of territory ISIS lost is significant.” How is that factual rather than opinion?
I got all the questions right except that one, according to Pew. I agree with Tired that this question is getting graded incorrectly. I’m not sure that invalidates the study, though having ten percent of the questions graded incorrectly is… well… significant. In my opinion.
By the way, in science it’s common to refer to a numerical result as “statistically significant” if the probability that it would have been that strong just by chance is below 0.05 (or sometimes 0.1). But I don’t remember that getting shortened to just “significant” and still being taken as a factual statement.
I’m inclined to lean on the "factual’ side, though I do see the OP’s point. “A significant portion of its territory” refers to the significance of the portion, not the significance of the territory. I’d under stand the statement to refer to an amount of the territory which is a significant amount, relative to the totality of the territory.
Obviously, there’s still a degree of judgment involved in deciding what’s a significant amount, relative to the whole. But where, as is in fact the case, Isis has lost virtually the whole of its territory, the statement isn’t really asking you to make a judgment about signficance. It’s asking you to agree or disagree that Isis has lost a large or substantial amount of the territory that it formerly held.
