Note: I know that there are real-world scenarios that parallel this, but I’m going to keep it somewhat abstract, so as to avoid arguing over how parallel the real-world cases are.
Suppose, for instance, we have a legislature composed of two parties. Every legislator can be given a one-dimensional ideology score from (say) 0 to 100, and every legislator with a score under 50 is a member of the liberal party, and every legislator with a score over 50 is a conservative.
Well, let’s say that the most liberal member of the conservative party, with a score a little over 50, has a shift in his ideology for some reason, so it goes to just under 50, and accordingly changes his official party affiliation. This is a win for the liberals, right?
But on the other hand, suppose we look at the average ideology of each of the parties. The conservative party has lost this moderate who was dragging their average down, so the conservative party on average has gotten more conservative. On the other hand, the liberal party has gained a moderate who’s now pulling their average up, so the liberal party has also, by this move, gotten more conservative.
So even though the average ideology of the legislature as a whole has become more liberal, the average ideology of each of the two parties (which, together, comprise the entire legislature) has gotten more conservative.
Has anyone looked at peculiarities like this before?
Pulling statistics out of subsets can result in all sorts of strange results. It’s one reason for the saying about lies and statistics. I don’t know what the name for this is.
In this case, the liberal and conservative subsets before and after the shift are different, so the before-after comparisons are comparing apples to oranges. It’s akin to the twins paradox in special relativity–there’s an implicit frame shift that’s important.
Indeed, the reverse is true as well: combining two sets, which individually exhibit one trend, can result in a set that exhibits the opposite trend. Statisticians know this effect as Simpson’s paradox.
I believe it was referring originally about migrants from Oklahoma to California and was stated in that respect by Will Rogers. It’s now known as the Will Rogers Phenomenon.
In case biqu’s mention didn’t make it clear, the OP’s paradox and Simpson’s paradox are essentially the same phenomenon: given variables A, B, and a weighted combination C of A and B, and variables A’ and B’ and a weighted combination C’ of A’ and B’, it’s possible to have A < A’ and B < B’ but have C > C’, so long as the weightings used in C’ are different from those in C.
[in the OP’s case, take A to be the initial average score of the liberals, B to be the initial average score of the conservatives, C to be the initial average score of the legislature as a whole, and A’, B’, C’ to be the finale average scores]
Actually, it looks to me like the Will Rogers Phenomenon is exactly the same phenomenon as the one I was referring to: Just substitute “conservativeness” for “intelligence”, “conservative party” for “Oklahomans”, “liberal party” for “Californians”, and “most liberal Conservative legislator” for “Okies”.
My concern was that “the Will Rogers phenomenon” might not be taken to specifically invoke the part of the OP where the average liberalness/intelligence of society as a whole goes up, in counter to what happens for each of the two states/parties. But, sure, you could work that into the Will Rogers setup as well. In which case, it would have the same similarity to Simpson’s paradox, but perhaps serve usefully to name the special case where the weightings are given by taking population-weighted averages, and the change in weightings is due to movement of population from one state/party to another.
It’s Simpson’s paradox (or closely related). Clicking Wikipedia just now I see Simpson’s and Will Rogers’ don’t link to each other. Am I overlooking some difference?
Er, I guess the usual Will Rogers phrasing has the intelligence of both parties go up, so that the paradoxical counter-effect, were one worked in, would be to have the intelligence of society go down. So, uh, change “up” to “up/down”. Whatever.
Anyway, yeah, septimus, the way I see it, the Will Rogers phenomenon is a special case of Simpson’s paradox.
Most paradox are based on less than definitive semantics, either in the initial premise, or the outcome. In your case, there is nothing illogical about the outcome. The average ‘conservativeness’ of the entire legislature was lowered. Your parenthetical comment ‘which, together, comprise the entire legislature’ implies a false correlation between the average of the entire body and the individual averages of the factions. Also, what if your ‘flip-flopper’ was the only legislator with a score over 50? I’m bad at math, but I think the average for conservatives would change to a divide by 0 error in that case.
To be clear, I know that it’s not a real paradox, since by definition, there’s no such thing. And I understand why it really is good news for the liberal party in my example: The liberal party now holds more power than they did before, and the conservative party now holds less.
That said, it’s not hard to construct a situation where it really would be relevant in the real world. If we stick with political ideology instead of intelligence, but make it people moving across state lines instead of party lines like in Will Rogers’ example, then you could, for instance, see both Oklahoma and California electing more conservative senators. In this case, it wouldn’t be relevant that California, by virtue of the move, has gained population and Oklahoma has lost it, since every state has the same number of senators regardless of population.
In fact, something quite like this happened in 2006, when the Democrats took control of Congress. I seem to remember their being a whole sub-genre of commentary that pointed out that the Democrats gained their majority by adding a bunch of conservative Democrats, and that this was somehow an indication that the country was still fundamentally center-right. Said commentary, of course, ignored the fact that Congress as a whole moved to the left.
OK, I’ve no argument with that. Did I get the math right? If so, I think you need an additional qualifier in the premise. And you might want to use ‘less than or equal’ or ‘greater than or equal’ to avoid issues with legislators whose score is exactly 50. IANA statistician. Maybe there are conventions for dealing with those issues already.