Statistics Question: Which statistical test to use?

Factual Questions
1

I am trying to help a friend with the statistical part of his thesis. I have tried to research this myself but I have to admit I am really not that good at statistics. Because he is under a tight time constraint, I thought I would ask you smart people to help me out.
He is doing a research on a small forum; testing the relationships between different characteristics of the posts and the number of replies.

For example: There are about 300 posts in total in this forum. These posts are put into 2 to 4 different categories depending on their characteristics. There would be uneven number of posts in each category.

He wants to test if a particular characteristic would result in more replies than the others.
So how could I set it up?

Test the mean of replies in each category against the population mean? Determine if each category is significantly different from the population mean?

Any way I could test that one category is significantly the largest, another is the second largest, another is third largest and the last one is the smallest?

AND most important: which test statistic to use? There seems to be a ton of them.
Any pointer would greatly be appreciated. Thank you all so much in advance.

2

If he’s just interested in looking at differences among groups, ANOVA is probably the way to go. If the groups are determined based on combinations of underlying characteristics, some kind of factorial design methods might be more appropriate. Neter et al.'s “Applied Linear Statistical Models” covers these topics in detail.

3

What about a chi-square test?

You would count the number of posts that fall into each category and then fill in a contingency table. There are several free programs on-line that will calculate the p-value (you will be testing the hypothesis that there are an equal number of replies for each category).

Hope this helps.

4

Thank you, ultrafilter and monstro. It’s really helpful to at least have some direction.

The following is exactly what my friend wanted to do.

There are 354 posts in this forum, averaging 2.32 replies per post.

The posts are put into 5 categories depending on who created the post:

_____# of posts in this category Average # of replies
Category 1________48________________________3.39
Category 2________41________________________1.68
Category 3________42________________________1.67
Category 4_______214________________________2.37
Category 5_________9________________________1.55
All I want is to statistically test and say that the type of posters in category 1 can generate more replies; while posters in category 2, 3 and 5 would generate fewer replies.

If this is too hard, could I at least prove that who made the post would at least cause a statistical difference from the population mean?
One big problem I found is that the # of replies is NOT normally distributed. It is closer to exponential distribution like this:
http://en.wikipedia.org/wiki/Image:Exponential_distribution_pdf.png
So I don’t know if I could even use ANOVA….

5

Chi-square is a non parametric test, so no worries there.

However, looking at your table …

the 354 posts comprise the population and each post is put into only one category. Thus you have have independent samples from the population – could try a Mann-Whitney U? This tests if the probablity distributions of the population and the sample are the same or not. You can specify the direction to see if a sample is higher.

6

Instead of working with averages, I’d recommend treating each post as an observation. So you’d be working with the number of replies for each post, rather than a summary statistic for each category.

So let’s say you have 48 posts that fit category A. Category A will then have a sample size of 48 (average equal to 3.39). Category B will have a sample size of 41. So on and so forth.

You’re now set up to do a Kruskal Wallis test, the nonparametric version of ANOVA. Instead of averages, it works on medians. I recommend a nonparametric test since, as you say, the data are not normal. You could transform them, but 1) that’s a pain and 2) that may not fix anything.

Mann-Whitney only works on two samples.

Chi-square would be good if you had another level of parsing (like the gender of the posters).

7

You might try a log transformation to get the data to look more normal. That’s done with count data a lot. If you have categories with 0 counts, you can add a constant to all the data.

Then, an analysis of varianve (ANOVA) would address a hypothesis test like

“the mean number of responses is equal for all 5 categories”

versus

“the mean number of responses varies for at least one category”.

From there, you can start setting up contrasts to see if the mean from one group is greater than the mean from another group, or all other groups combined.

You can look up “contrasts” or “comparisons” in experimental design books.

8

[QUOTE=Trunk]
You might try a log transformation to get the data to look more normal. That’s done with count data a lot.
[/QUOTE]

It’s my understanding from coursework that square root transform is usually used for count (Poisson) data.

9

[QUOTE=iwakura43]
It’s my understanding from coursework that square root transform is usually used for count (Poisson) data.
[/QUOTE]

TRY BOTH!!!

Seriously.

They both can be used for count data, and you can plot the results against a normal distribution to see which one fits better.

10

Thank you all so much again! I knew I could count on you guys! There is noway I could have figured this out without your help, especially in such a short time!

I came across this wonderful thing called SPSS.
This is what I understand so far:
In SPSS, run a Kruskal-Wallis test first to check if the “category of poster” affects “# of replies”.
Analyze -> Nonparametric Tests -> K Independent Sample
Use “# of replies” as Test Variable; use “category of poster” as Grouping Variable.
Output: Asymp. Sig: .032 -> so “category of poster” statistically affects “# of replies”

Is this right so far?
Could I then use Mann-Whitney:
Nonparametric Tests -> 2 Independent Samples
for each pair of “category of poster” to find out if one category has higher # of replies to another?
So, a Mann-Whitney for between Category 1&2, 1&3, 1&4, 1&5, 2&3, 2&4, 2&5, 3&4, 3&5, 4&5?
I do need other levels of parsing (yes, gender of posters happens to one of them!).
If I want to find out if a combination of characteristics would result in more replies, do I use:
Descriptive Statistics -> Crosstabs -> then use Chi-Square for this?
Is there any reason that I have to transform the data into normal distribution instead of just using nonparametric tests?

11

God, the more you read the more problem you run into… hehehe… :smack:

It seems that crosstabs chi-square is not suitable for “# of replies” as it would have too many categories. So I couldn’t use it for test the effect of a combination of characteristics?

Is there a non-parametric statistical test for factorial design analysis?

Or do I have to use Univariate Analysis?? Which, as it happens, need normal distribution?

Is this why everyone keep telling me to transform data into normal distribution.

12

[QUOTE=PReed]
Thank you all so much again! I knew I could count on you guys! There is noway I could have figured this out without your help, especially in such a short time!

I came across this wonderful thing called SPSS.
This is what I understand so far:
In SPSS, run a Kruskal-Wallis test first to check if the “category of poster” affects “# of replies”.
Analyze -> Nonparametric Tests -> K Independent Sample
Use “# of replies” as Test Variable; use “category of poster” as Grouping Variable.
Output: Asymp. Sig: .032 -> so “category of poster” statistically affects “# of replies”

Is this right so far?
Could I then use Mann-Whitney:
Nonparametric Tests -> 2 Independent Samples
for each pair of “category of poster” to find out if one category has higher # of replies to another?
So, a Mann-Whitney for between Category 1&2, 1&3, 1&4, 1&5, 2&3, 2&4, 2&5, 3&4, 3&5, 4&5?
I do need other levels of parsing (yes, gender of posters happens to one of them!).
If I want to find out if a combination of characteristics would result in more replies, do I use:
Descriptive Statistics -> Crosstabs -> then use Chi-Square for this?
Is there any reason that I have to transform the data into normal distribution instead of just using nonparametric tests?
[/QUOTE]

You don’t want to pick and choose which test to use like this. In theory, you should design an experiment with the test statistic clearly in mind.

I’ve never done a chi-square in spss. I usually just wimp out and use the free contingency tables that are available on-line. So you’d set up the data like this:

CAT________1___2__3
Male________xx xx xx

Female______xx xx xx
where xx equals total number of replies (not average).

You could also consider a two-way Kruskal Wallis (don’t know if spss has this function). But since you have vastly different sample sizes per category (9 versus 214), you probably shouldn’t be doing a KW at all. If I were your fried, I wouldn’t use category 5. I would either lump it in with a similar category, or I would drop it completely. Such a low sample size (relative to the others) is going to skew your results. And likewise, category 4 is too large to fairly compare to the others. You shouldn’t have one sample that’s less than half the size of the others.

Yes, you can use the Mann-Whitney as a post-hoc test after a KW test.
For a chi-square, you could reduce the table to a 2X2 (by leaving out categories) until you find the differences. But what most people do is just eyeball the table and guess at the ones that look different. Not as rigorous, but it generally works.

As the study design stands now, I’m leaning more towards the chi-square than the Kruskal Wallis.

I wouldn’t bother with transformation. Nonparametric tests are just as good as parametric tests. In fact, depending on the nature of the data, they can be more powerful. People tend to stay away from them out of fear of the unknown (kind of like how we treat lesser known name brands at the grocery store), but this an irrational fear.

13

Also, I don’t know if you know this, but the cool thing about nonparametric tests is that it doesn’t matter if you transform them. You will get the same result no matter what you do to the data.

14

[QUOTE=PReed]
God, the more you read the more problem you run into… hehehe… :smack:

It seems that crosstabs chi-square is not suitable for “# of replies” as it would have too many categories. So I couldn’t use it for test the effect of a combination of characteristics?
[/quote]

Are you doing total number of replies per category?

15

Thank god for Straight Dope Forum, and thank god for you, monstro!

Sorry, I was a little bit confused about the chi-square test just now.

From you last posts, I see exactly what to do now. Thank you especially for pointing out that I should use total number of replies, I would probably have used mean instead.

Let me try this and see how it goes. :slight_smile: