# Statistics: Can someone help with this analysis of percentage drop in percentages?

**URL:** <https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548>\
**Category:** Factual Questions\
**Created:** [February 28, 2014, 9:53pm UTC](https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548 "2014-02-28T21:53:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Leo\_Bloom](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/leo_bloom/32/10377_2.png) [@Leo\_Bloom](https://boards.straightdope.com/u/Leo_Bloom)\
**Post date:** [February 28, 2014, 9:53pm UTC](https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548/1 "2014-02-28T21:53:09Z")

</div>

Can anyone comment on the _mathematical veracity of the argument_ in [todays’s "The Best of the Web"](http://m.us.wsj.com/articles/SB10001424052702304026804579411110505865626?mobile=y)column at the WSJ, discussing a [NYTimes story](http://www.nytimes.com/2014/02/26/health/obesity-rate-for-young-children-plummets-43-in-a-decade.html?_r=0)earlier this week:  
[…A NYTimes] news story from earlier in the week [is] headlined “Obesity Rate for Young Children Plummets 43% in a Decade.” But if you read closely, you’ll see the headline oversells the good news:  
[INDENT]The figures on Tuesday showed a sharp fall in obesity rates among all 2- to 5-year-olds, offering the first clear evidence that America’s youngest children have turned a corner in the obesity epidemic. About 8 percent of 2- to 5-year-olds were obese in 2012, down from 14 percent in 2004.  
It’s true that 8% is 43% less than 14%. But percentages of percentages are tricky. The decline amounts to only six percentage points, or 6% of the total population in that age range. And it’s a narrow age range, raising the possibility that the findings are the result of sampling error or short-term demographic trends. (Many children who were 2 to 5 in 2012 were born during a recession, and recessions tend to depress fertility, especially among the less affluent.)[/INDENT]

---

<div class="post-metadata">

**Author:** ![dasmoocher](https://avatars.discourse-cdn.com/v4/letter/d/dbc845/32.png) [@dasmoocher](https://boards.straightdope.com/u/dasmoocher)\
**Post date:** [February 28, 2014, 10:07pm UTC](https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548/2 "2014-02-28T22:07:19Z")

</div>

If you really want to oversell it, couldn’t you say that there were 75% more overweight children in 2004 than 2012? 😉

---

<div class="post-metadata">

**Author:** ![Measure\_for\_Measure](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/measure_for_measure/32/557_2.png) [@Measure\_for\_Measure](https://boards.straightdope.com/u/Measure_for_Measure)\
**Post date:** [March 1, 2014, 2:57am UTC](https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548/3 "2014-03-01T02:57:25Z")

</div>

Let’s say the population was the same at the beginning and the end of the decade, for simplicity. Consider the following:

```auto

Obesity2/Pop Obesity2
--------------- -1 = --------------- -1 = % drop in obesity. 
Obesity1/Pop Obesity1

```

I may have mangled that a little, but you can see that the figure isn’t total garbage.  
The report by the NYT is highly dubious though. It looks like they have emphasized an outlying datapoint, given that obesity among the age group immediately above it hasn’t changed much. Look even closer and you see that they rely heavily on the most recent data at the end of the decade. Caveat lector.

> **[Obesity drop among 2-5 year-olds is even more baffling than I thought](https://www.motherjones.com/kevin-drum/2014/02/obesity-drop-among-2-5-year-olds-even-more-baffling-i-thought/)**
>
> It appears to have precisely zero effect as children get older.

---

<div class="post-metadata">

**Author:** ![lazybratsche](https://avatars.discourse-cdn.com/v4/letter/l/ba8739/32.png) [@lazybratsche](https://boards.straightdope.com/u/lazybratsche)\
**Post date:** [March 1, 2014, 1:58pm UTC](https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548/4 "2014-03-01T13:58:12Z")

</div>

I just skimmed [the actual paper](https://jama.jamanetwork.com/article.aspx?articleid=1832542) and I get a whiff of “data dredging”. Here’s the relevant passage in the paper for this result:

> [@](#):
>
> Results of linear unadjusted trend tests between 2003-2004 and 2011-2012 are shown in Table 6. Among infants and toddlers from birth to aged 2 years, there was no significant change in high weight for length prevalence (−1.4 percentage points; P = .72). Among children and adolescents aged 2 to 19 years, there was no significant change overall (−0.2 percentage points; P = .78), but **there was a significant decrease in obesity prevalence among 2- to 5-year-old children (−5.5 percentage points; P = .03)**. Among adults, there was no significant change in obesity prevalence in the total population (+2.8 percentage points; P = .09), but there was an increase in prevalence among adults aged 60 years and older (+4.4 percentage points; P = .004).

The statistical test in this case was a bunch of pairwise T-tests, with no adjustment for multiple comparisons as far as I can tell. This is a naiive and often problematic approach, especially for post-hoc data analysis.

The P values tell you the probability of getting your observation from two random samples from the same group. P = 0.05, the de-facto threshold for “statistical significance”, means that there is a 1 in 20 chance that your observed difference is just a random fluke. So if you make lots of different comparisons at the same time, you should expect that some will be “significant” by chance alone. (Illustrated by this [XKCD](https://xkcd.com/882/).) Here, the authors of the paper report _nine statistical comparisons, where only one is “significant”_.

It is possible that the authors may have made a lot of other unreported tests, between the 2011-2012 period and each of the four previous time periods, for each age group, adding up to 36 comparisons in total. If they made all of those comparisons, of course a few are going to be “statistically significant” by chance. This practice is a statistical sin but distressingly common in the medical literature.

Caveats: I am not an actual biostatistician. I am currently trying to analyze a bunch of old datasets where there are no established multiple-comparison post-hoc methods. That means I probably know just enough dangerous.

TL;DR: I bet that this results is just a fluke.

---

<div class="post-metadata">

**Author:** ![Leo\_Bloom](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/leo_bloom/32/10377_2.png) [@Leo\_Bloom](https://boards.straightdope.com/u/Leo_Bloom)\
**Post date:** [March 1, 2014, 5:15pm UTC](https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548/5 "2014-03-01T17:15:12Z")

</div>

Believe it or not, I’m sure I know the xkcd strip you cite :::checks::. Yup.

Thanks to all. It’s amusing that a columnist from _Mother Jones_ and one for the _WSJ_ both picked up on that and called it dicey. And tells you something about the implacable nature of math.

---

<div class="post-metadata">

**Author:** ![DataX](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@DataX](https://boards.straightdope.com/u/DataX)\
**Post date:** [March 1, 2014, 5:49pm UTC](https://boards.straightdope.com/t/statistics-can-someone-help-with-this-analysis-of-percentage-drop-in-percentages/682548/6 "2014-03-01T17:49:23Z")

</div>

> [@lazybratsche](#):
>
> I just skimmed [the actual paper](https://jama.jamanetwork.com/article.aspx?articleid=1832542) and I get a whiff of “data dredging”. Here’s the relevant passage in the paper for this result:
> 
> The statistical test in this case was a bunch of pairwise T-tests, with no adjustment for multiple comparisons as far as I can tell. This is a naiive and often problematic approach, especially for post-hoc data analysis.
> 
> The P values tell you the probability of getting your observation from two random samples from the same group. P = 0.05, the de-facto threshold for “statistical significance”, means that there is a 1 in 20 chance that your observed difference is just a random fluke. So if you make lots of different comparisons at the same time, you should expect that some will be “significant” by chance alone. (Illustrated by this [XKCD](https://xkcd.com/882/).) Here, the authors of the paper report _nine statistical comparisons, where only one is “significant”_.
> 
> It is possible that the authors may have made a lot of other unreported tests, between the 2011-2012 period and each of the four previous time periods, for each age group, adding up to 36 comparisons in total. If they made all of those comparisons, of course a few are going to be “statistically significant” by chance. This practice is a statistical sin but distressingly common in the medical literature.
> 
> Caveats: I am not an actual biostatistician. I am currently trying to analyze a bunch of old datasets where there are no established multiple-comparison post-hoc methods. That means I probably know just enough dangerous.
> 
> TL;DR: I bet that this results is just a fluke.

I can’t seem to get the JAMA reader to work on my iPad (it’s stuck on the last article I read - so your link just sends me to that).

Anyway - it appears you are right - as one of the comments in one of the articles on Mother Jones quoting the original study - they did NOT correct for the multiple comparisons.

That is actually a HUGE drop on first glance - I would have guessed with the CDC - they’d be dealing with a lot of data. But a p of 0.03 isn’t that impressive to begin with even if there weren’t multiple comparisons - as you have the same issue on a different scale. Of every 20 experiments you do at random testing one thing - you’ll get a “significant” result one time. Obviously more papers are sent in with significant results - and you end up with claims like this. Usually the authors throw in some caution in the paper somewhere (they did here as well if you can believe the quote in the comments section). The media then often report - only sometime with the caution included (of course almost never in the headline - or first paragraph - which if someone is browsing the news - is all they read).
