Flawed Metrics and the Politics of Perception

Great Debates
1

In this thread, astorian made a claim–that modern Supreme Court justices often turn out to be more liberal than expected–which, upon inspection, proved almost entirely ill-founded. To challenge his assertion, I used the following data:

The degree to which the justices were perceived to be liberal upon nomination, derived from a study in the American Journal of Political Science, and

The degree to which their later voting behavior conformed to these initial perceptions, as examined by Congressional Quarterly.

Basically, I appealed to authority, citing relevant and reputable sources in order to settle the argument. After all, with such a large body of scholarship existing in the social sciences, there’s a good chance someone’s already done the research on any given topic; all we have to do is find it and apply it.

But this got me thinking: it’s staggering just how much we rely on other people’s numbers in modern society. National policy and public perceptions are shaped heavily by the statistics we see and hear, and we must take it on trust that these statistics, and the metrics which engender them, are valid and accurate. What if the methodology of the studies I’d cited had been severely flawed? There’d be no way for me to know, short of crunching all the numbers myself and, in essence, replicating the research. More importantly, what if some of the most fundamental numbers in our society–numbers used faithfully on the national news, on the floor of the Senate, and in boardrooms across America to provide a snapshot of current conditions–are themselves the product of severely faulty metrics?

I submit that at least four key economic indicators measure far less than we are given to assume–specifically, that U.S. poverty, unemployment, consumer confidence, and television viewership are not accurately reflected by their respective indexes, and that policy and perception in these areas are therefore badly misguided:

Poverty: The official U.S. poverty level is calculated by assuming that a family of four spends one-third of its total income on food, and then determining the cost of an annual food budget which provides minimum nutritional requirements. Because the calculations are based on 1950s numbers–that is, families today spend much less of their income on food (one-fifth or one-sixth), and are often unable to buy food as cheaply as is assumed in the original study–they significantly underestimate the number of people living in poverty. In fact, if the rates were recalculated using a more accurate allocation of family income, the incidence of American poverty in 1988 would have almost doubled the “official” rate of that time, from 13 percent to 25.8 percent of the population. The full-time living wage necessary to subsist above the adjusted level without government assistance was $9.62 an hour in 1990–far above any minimum wage in the United States today.

Unemployment: The official unemployment rate underestimates the true number of unemployed even more grievously. Aside from the fact that conventional economists’ definition of “full employment”–the number of unemployed necessary and natural in a market economy–has been consistently revised upwards, from 3 percent in the 1950s to 6 percent in the 1990s, the exclusion of discouraged workers from the unemployment count and the shifting of millions of involuntary part-time workers into the ranks of the “employed” has led to numbers quite divorced from reality. Also, the link between unemployment and crime means that more and more people will end up in prison–and not counted as officially unemployed–as jobs are lost. This doesn’t even get into the relative value of jobs being created: real hourly wages have decreased since 1973, and trends show a rapidly declining manufacturing sector bolstered only partially by an expanded service sector–low-wage, low-skill jobs–and the rise of a small and educated “information elite” for technology jobs.

Consumer Confidence: The consumer confidence index is a metric which purports to reveal how consumers are feeling about current economic conditions. It’s computed by a private business, The Conference Board Inc., which, in Jim Hightower’s description, is “a service company for corporate executives, holding seminars and conferences for them, doing management research, organizing executive ‘networking groups,’ and the like.” Its survey consists of sixteen questions–mostly concerning marketing–sent to five thousand people every month from an annually selected, unrepresentative pool of 700,000. Further, there are questions about the tabulations themselves–Al Sindlinger, the pollster who devised the term “consumer confidence” in 1928, stopped contracting with The Conference Board because he felt they were suppressing results that did not present a positive outlook. According to Sindlinger, “It’s no longer a measurement, but a promotion.” Yet the consumer confidence index is reported in publications like The Wall Street Journal as an accurate reflection oof the country’s economic mood.

The Nielsens: Boy, where to start? Market share and ratings points are determined by various mutable incarnations of the Nielsen system. These vital statistics supposedly track viewer behavior accurately enough to rationally dictate programming choices and advertising revenue. The sample sizes in the methodological wonderlands are sometimes so small (for cable channels, for example) that three or four households can mean the loss or gain of fifty million dollars. More comprehensively, the system was never intended (by its creator, A. C. Nielsen) “to be construed as a finite measurement of the viewing habits of millions.” (Fred Friendly, Due to Circumstances Beyond Our Control) Friendly goes on to say that a television industry who relies so heavily on the Nielsens–and the American TV industry relies almost exclusively on the Nielsens and similar metrics in computing market share–“is in the position of gauging space-age tolerances with the kind of dip stick used to measure the amount of gas in a Model T.” From the self-selected sample of Nielsen homes, programming for the entire population is determined.

I guess what I’m saying is this: There’s no way for us to really know if the numbers we’re hearing are accurately, or even objectively, derived. I think there needs to be a fuller understanding by the public at large of the metrics that are used to shape policy–we tend to assume, understandably, that if we’re told that unemployment is 4.5 percent, it means that 4.5 percent of American adults are without jobs. That ain’t correct. But often, it’s not what’s correct that matters…it’s whatever is said with the greatest conviction, or repeated the most times. Perception shapes reality.

Thoughts?

2

Well, I could bite off everything you’ve addressed, but then I’d look pretty silly with a mouthful of unchewable subject matter. So I’ll cut off a small slice and have at it.

I think that a flawed, overly narrow, biased measurement can still have its uses. Look for example at unemployment - if the unemployment rate is calculated consistently and precisely, we can still use it to measure economic changes. It will prove useful even if it’s not measuring anything in particular. What is the true number of unemployed people? Hard to say, but if the nominal unemployment rate doubled, I’d bet good money that the true number of unemployed people went up substantially as well.

I’d say the same for the poverty rate. It’s hard to agree on how many people are poor, but if statistics from a source we trust tell us that children are more likely to be poor than retirees, then at least we know where to direct our efforts. I’m not saying we all trust the same sources, or that the AARP is going to start asking to redirect money from Medicare to WIC, I’m just saying, flawed tools are bettter than no tools.

Some flawed metrics do create real problems; I for one believe that inflation has been overestimated for years. This has caused Cost Of Living Adjustments to be too high, which has in turn caused entitlements that have COLAs to be growing at the expense of those programs that don’t. So I agree with you in a lot of ways. We should be constantly striving to improve both the accuracy and the precision of our measurements, without abandoning the old ones. After all, the old ones, warts and all, are all we are going to have to compare ourselves with the past.

What upsets me the most are the measurements which weren’t intended to be accurate, which aren’t widely accepted by social scientists, which capture the public’s imagination because they conform to what the public wants to believe. I wish I could think of some off-hand, but I usually find them so annoying that I try to forget them as soon as possible. The best one I can think of is “most people who have been hypnotized report that they didn’t know they were hypnotized”, which people seem to think means that they had no recollection of the session at all (in reality it means that they remember everything that had transpired, only they would describe it as a normal conversation, not an altered state of consciousness). Anyway, I’m going to start having trouble chewing if I continue.

3

Gadarene said: *But often, it’s not what’s correct that matters…it’s whatever is said with the greatest conviction, or repeated the most times. Perception shapes reality. *

Well, I hope you’re not looking for a lot of argument on that one! :slight_smile: I guess what you want to debate is, how fair are the numbers and how can we know? (And why the Sam Hill didn’t you give us any cites, in a thread that’s all about how unreliable other people’s statistics may be? ;)) Piece by piece:

*Poverty: The official U.S. poverty level is calculated by assuming that a family of four spends one-third of its total income on food, and then determining the cost of an annual food budget which provides minimum nutritional requirements. Because the calculations are based on 1950s numbers–that is, families today spend much less of their income on food (one-fifth or one-sixth), and are often unable to buy food as cheaply as is assumed in the original study–they significantly underestimate the number of people living in poverty. *

Judging from this Health and Human Services document,“History of the Poverty Thresholds”, this is a fair description of how the poverty level determinations originated. It says that “in 1969, the [interagency Poverty Level Review] Committee decided that the thresholds would be indexed by the Consumer Price Index instead of by the per capita cost of the economy food plan, and that farm poverty thresholds would be set at 85 percent rather than 70 percent of corresponding nonfarm thresholds.” The metric doesn’t seem to have changed significantly since then except that in 1981, “the farm/nonfarm differential was eliminated by applying nonfarm poverty thresholds to all families.”

*In fact, if the rates were recalculated using a more accurate allocation of family income, the incidence of American poverty in 1988 would have almost doubled the “official” rate of that time, from 13 percent to 25.8 percent of the population. *

Now here is where I’d especially like to see some cites and explanation of the numbers provided. I agree that poverty is probably more pervasive than the official numbers indicate, but over a quarter of the entire population? By what standards? I found a Census Bureau report called “Examining experimental poverty measures” that looks at different metrics, particularly those provided in the 1995 report from the National Academy of Sciences (NAS) Panel on Poverty and Family Assistance, Measuring Poverty: a New Approach. This document is pretty sticky going, but it describes the basic thrust of the NAS report as follows:

This sounds like the sort of recalculation of the metric that you’re talking about. But as far as I can see from the given figures, the poverty incidence in 1990 (the earliest year they include) according to the NAS experimental measures would have been 16.1% as opposed to 13.5% by the official standard. (Other experimental measures discussed give percentages up to 17.something%, but that is nothing like the 25.8% of the entire population you mention.) Obviously there are radical differences of opinion and/or approach here; where are your numbers coming from?

*The full-time living wage necessary to subsist above the adjusted level without government assistance was $9.62 an hour in 1990–far above any minimum wage in the United States today. *

I’m an ardent supporter of the living-wage movement, but I have to ask: subsist where? The biggest problem with this number seems to be that there’s only one of it, whereas we know that the cost of living varies widely from place to place. According to this 1998 FIU Center for Labor Research and Studies report, when Baltimore instituted a living-wage ordinance in 1994, the mandated minimum was $6.10 an hour, increasing to $7.70 by 1999. Proposed living-wage levels generally seem to use a yardstick of 10% above the official poverty level, but according to this report, even 20% above the 1997 poverty level would mean an hourly wage of only $9.34 for the Floridians in question. You may be arguing that these are underestimates, or still require government assistance for the living-wage workers, which may be very true, but what’s the claim and what’s the evidence?

Man, that was just one of the pieces and I’m already out of time on this. Produce your evidence and let’s duke it out! :slight_smile:

4

Kimstu:

The numbers on the recalculated poverty rate and the subsistence wage come from the 1993 edition of The State of Working America by Lawrence Mishel and Jared Bernstein of the Economic Policy Institute, an admittedly left-leaning organization. On second blush, I’d say your data is likely to be more accurate–or more current, at any rate. :slight_smile: Mishel and Bernstein publish a new edition annually; I’ll see if I can find the most current and see what it says.

As for cites on the rest: the unemployment information came from a variety of sources, most notably Stanley Aronowitz’s The Jobless Future (1995), Michael Yates’s Longer Hours, Fewer Jobs: Employment and Unemployment in the United States (1994), Jeremy Rifkin’s The End of Work (1995), and good ol’ boy Jim Hightower and his various populist rantings. grin

Hightower was the main source for the stuff on the CCI (in his book, If the Gods Had Meant For Us To Vote, They Would Have Given Us Candidates) , though the information is objectively verifiable. As a side note; do a search for “consumer confidence index” on Google and you’ll see the extent to which it’s relied upon as a valid indicator of economic conditions. Scary, really.

And the Nielsens: I guess I didn’t provide much quantifiable evidence on that one. Here’s an essay that backs me up, though. This is a good article from Brill’s Content about the character of the Nielsens ratings. Anectodally, I’ve talked with my roommate, senior director at the local ABC affiliate, at length about his perceptions of the ratings, and perused industry mags like Electronic Media and Broadcasting & Cable. (Also, I wholeheartedly recommend Fred Friendly’s book to everyone–it’s a fascinating account of the transformation of television into a predominantly for-profit enterprise, and the resultant conflict with news reporting.)

Does that help at all?

Boris B: I do agree that there’s value in some of the metrics despite their limitations; I just think it’s important that the limitations be recognized as such.

Interesting stuff from both of you; lemme think about it (cop out! :)) and get back to ya further.

5

As Mark Twain said - “There are three kinds of lies: lies, damned lies and statistics” and “Get your facts first, and then you can distort them as much as you please.”

Question every poll and statistic result given you. It’s fascinating how easy it is to sway the results of a poll by the wording of the question, the order in which they’re asked, and even the tone of voice used.

I don’t believe in snapshot analysis of data. For example, I don’t believe you can accurately measure unemployment data NOW. As Boris B said, it’s much more useful to see trends as long as the methods and measures remain consistent. Even then, environments change, economies change, politics change, etc.